last last committed update 00010000:00080000:3.0:1524592148.906276:0:8130:0:(ldlm_lib.c:2492:target_committed_to_req()) 061cd4a0-cd22-5048-946a-d8f4ec53917a last_committed 2937774893830, transno 2937774897173, xid 1598582492444016 ... 00000100:00100000:3.0:1524592151.996038:0:18109:0:(service.c:1955:ptlrpc_server_handle_req_in()) got req x1598582492486880 00000100:00100000:3.0:1524592151.996042:0:18109:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-54@gni, seq: 1564278792 00000100:00100000:3.0:1524592151.996043:0:18109:0:(service.c:2103:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc mdt01_003:061cd4a0-cd22-5048-946a-d8f4ec53917a+6881:15972:x1598582492486880:12345-54@gni:101 ... first ENODEV 00000100:00080000:1.0:1524592151.996485:0:2924:0:(niobuf.c:583:ptlrpc_send_reply()) sending ENODEV from failed obd 16 last committed update - aka sync. 00010000:00080000:3.0:1524592151.996589:0:5591:0:(ldlm_lib.c:2492:target_committed_to_req()) 0c6ce748-6618-ff9d-ad84-6e6f761558b2 last_committed 2937774893859, transno 2937774968804, xid 1598584386635888 --- last on disk 00000004:00080000:13.0:1524592151.996630:0:8166:0:(osp_precreate.c:1476:osp_precreate_get_fid()) next [0x240000400:0x91561aae:0x0] last created [0x240000400:0x91562c9d:0x0] 00000004:00080000:13.0:1524592151.996632:0:8166:0:(osp_object.c:1547:osp_object_create()) last fid for osp_object is [0x240000400:0x91561aae:0x0] 00000004:00080000:13.0:1524592151.996867:0:8166:0:(osp_object.c:1601:osp_object_create()) snx11205-OST0000-osc-MDT0001: Wrote last used FID: [0x240000400:0x91561aae:0x0], index 0: 0 -- requested create 00000004:00080000:3.0:1524592151.996637:0:18109:0:(osp_precreate.c:1476:osp_precreate_get_fid()) next [0x240000400:0x91561aaf:0x0] last created [0x240000400:0x91562c9d:0x0] 00000004:00080000:3.0:1524592151.996639:0:18109:0:(osp_object.c:1547:osp_object_create()) last fid for osp_object is [0x240000400:0x91561aaf:0x0] 00000004:00080000:3.0:1524592151.996641:0:18109:0:(osp_object.c:1601:osp_object_create()) snx11205-OST0000-osc-MDT0001: Wrote last used FID: [0x240000400:0x91561aaf:0x0], index 0: 0 -- 00000004:00080000:3.0:1524592151.996642:0:18109:0:(osp_precreate.c:1476:osp_precreate_get_fid()) next [0x2c0000400:0x919e3d36:0x0] last created [0x2c0000400:0x919e50fb:0x0] 00000004:00080000:3.0:1524592151.996643:0:18109:0:(osp_object.c:1547:osp_object_create()) last fid for osp_object is [0x2c0000400:0x919e3d36:0x0] 00000004:00080000:3.0:1524592151.996644:0:18109:0:(osp_object.c:1601:osp_object_create()) snx11205-OST0001-osc-MDT0001: Wrote last used FID: [0x2c0000400:0x919e3d36:0x0], index 1: 0 00000004:00080000:3.0:1524592151.996644:0:18109:0:(osp_precreate.c:1476:osp_precreate_get_fid()) next [0x280000400:0x90d60cab:0x0] last created [0x280000400:0x90d623e7:0x0] 00000004:00080000:3.0:1524592151.996645:0:18109:0:(osp_object.c:1547:osp_object_create()) last fid for osp_object is [0x280000400:0x90d60cab:0x0] 00000004:00080000:3.0:1524592151.996646:0:18109:0:(osp_object.c:1601:osp_object_create()) snx11205-OST0003-osc-MDT0001: Wrote last used FID: [0x280000400:0x90d60cab:0x0], index 3: 0 00000004:00080000:3.0:1524592151.996647:0:18109:0:(osp_precreate.c:1476:osp_precreate_get_fid()) next [0x200000400:0x90bdfade:0x0] last created [0x200000400:0x90be10d8:0x0] 00000004:00080000:3.0:1524592151.996648:0:18109:0:(osp_object.c:1547:osp_object_create()) last fid for osp_object is [0x200000400:0x90bdfade:0x0] 00000004:00080000:3.0:1524592151.996649:0:18109:0:(osp_object.c:1601:osp_object_create()) snx11205-OST0002-osc-MDT0001: Wrote last used FID: [0x200000400:0x90bdfade:0x0], index 2: 0 00000001:00080000:3.0:1524592151.996669:0:18109:0:(tgt_lastrcvd.c:1145:tgt_last_rcvd_update()) transno = 2937774968811, last_committed = 2937774893859 00000004:00080000:3.0:1524592151.996687:0:18109:0:(mdt_open.c:674:mdt_mfd_set_mode()) [0x340148dee:0x127df:0x0] Change mfd mode 0 -> 01102. 00000004:00080000:3.0:1524592151.996690:0:18109:0:(mdt_handler.c:2803:mdt_save_lock()) request = ffff880171f62a00 reply state = ffff880c67ba5000 transno = 2937774968811 00010000:00080000:3.0:1524592151.996696:0:18109:0:(ldlm_lib.c:2492:target_committed_to_req()) 061cd4a0-cd22-5048-946a-d8f4ec53917a last_committed 2937774893830, transno 2937774968811, xid 1598582492486880 00000100:00080000:3.0:1524592151.996697:0:18109:0:(niobuf.c:583:ptlrpc_send_reply()) sending ENODEV from failed obd 16 00000100:00100000:3.0:1524592151.996703:0:18109:0:(service.c:2153:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc mdt01_003:061cd4a0-cd22-5048-946a-d8f4ec53917a+6888:15972:x1598582492486880:12345-54@gni:101 Request procesed in 660us (669us total) trans 2937774968811 rc -19/-19 00000100:00100000:3.0:1524592151.996705:0:18109:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-54@gni, seq: 1564278792 -- fail -- restart 00000001:02080000:12.0:1524592174.240275:0:20073:0:(tgt_lastrcvd.c:1419:tgt_clients_data_init()) RCVRNG CLIENT uuid: 061cd4a0-cd22-5048-946a-d8f4ec53917a idx: 31 lr: 2916305857434 srv lr: 2937757630464 lx: 0 gen 767 00000001:00080000:9.0:1524592175.196027:0:20073:0:(tgt_lastrcvd.c:1912:tgt_reply_data_init()) snx11205-MDT0001: restore reply ffff88103870d420: xid 1598582492486864, transno 2937774968779, client gen 767, slot idx 40 00000001:00080000:9.0:1524592175.196133:0:20073:0:(tgt_lastrcvd.c:1912:tgt_reply_data_init()) snx11205-MDT0001: restore reply ffff880f988835a0: xid 1598582492486896, transno 2937774968802, client gen 767, slot idx 98 00000001:00080000:9.0:1524592175.196150:0:20073:0:(tgt_lastrcvd.c:1912:tgt_reply_data_init()) snx11205-MDT0001: restore reply ffff881032bd1f60: xid 1598582492486880, transno 2937774968811, client gen 767, slot idx 117 00010000:02080000:10.0:1524592182.602442:0:20319:0:(ldlm_lib.c:1174:target_handle_connect()) snx11205-MDT0001: connection from 061cd4a0-cd22-5048-946a-d8f4ec53917a@54@gni recovering/t2937774893877/2937774968811 exp ffff880f72086000 cur 1524592182 last 1524592174 00000100:02000000:10.0:1524592182.602458:0:20319:0:(import.c:1552:ptlrpc_import_recovery_state_machine()) snx11205-MDT0001: Connection restored to 061cd4a0-cd22-5048-946a-d8f4ec53917a (at 54@gni) 00010000:00080000:10.0:1524592182.602462:0:20319:0:(ldlm_lib.c:2492:target_committed_to_req()) 061cd4a0-cd22-5048-946a-d8f4ec53917a last_committed 2937774968811, transno 0, xid 1598582492487072 00000100:00100000:10.0:1524592182.602469:0:20319:0:(service.c:2153:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc mdt01_003:061cd4a0-cd22-5048-946a-d8f4ec53917a+5:3559:x1598582492487072:12345-54@gni:38 Request procesed in 35us (87us total) trans 0 rc 0/0 00000100:00100000:10.0:1524592182.654255:0:20319:0:(service.c:2103:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc mdt01_003:061cd4a0-cd22-5048-946a-d8f4ec53917a+5:3559:x1598582492487088:12345-54@gni:400 00010000:00080000:10.0:1524592182.654260:0:20319:0:(ldlm_lib.c:2359:target_queue_recovery_request()) @@@ queue final req req@ffff880f70d39b00 x1598582492487088/t0(0) o400->061cd4a0-cd22-5048-946a-d8f4ec53917a@54@gni:-1/-1 lens 224/0 e 0 to 0 dl 1524592237 ref 2 fl Interpret:H/c0/ffffffff rc 0/-1 00000100:00100000:10.0:1524592182.654265:0:20319:0:(service.c:2153:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc mdt01_003:061cd4a0-cd22-5048-946a-d8f4ec53917a+5:3559:x1598582492487088:12345-54@gni:400 Request procesed in 11us (75us total) trans 0 rc 0/-999 00010000:00080000:5.0:1524592183.842541:0:20038:0:(ldlm_lib.c:2414:target_queue_recovery_request()) Next recovery transno: 2937774724128, current: 2937761105239, replaying 00010000:00080000:5.0:1524592183.917746:0:20038:0:(ldlm_lib.c:2414:target_queue_recovery_request()) Next recovery transno: 2937774724128, current: 2937761105242, replaying 00010000:00080000:5.0:1524592183.917909:0:20038:0:(ldlm_lib.c:2414:target_queue_recovery_request()) Next recovery transno: 2937774724128, current: 2937766032569, replaying 00010000:00080000:13.0:1524592183.932552:0:20038:0:(ldlm_lib.c:2414:target_queue_recovery_request()) Next recovery transno: 2937774724128, current: 2937766034962, replaying 00010000:00080000:13.0:1524592183.932725:0:20038:0:(ldlm_lib.c:2414:target_queue_recovery_request()) Next recovery transno: 2937774724128, current: 2937766107292, replaying 00010000:00080000:13.0:1524592183.943923:0:20038:0:(ldlm_lib.c:2414:target_queue_recovery_request()) Next recovery transno: 2937774724128, current: 2937766109664, replaying 00010000:00080000:13.0:1524592183.944097:0:20038:0:(ldlm_lib.c:2414:target_queue_recovery_request()) Next recovery transno: 2937774724128, current: 2937774011747, replaying 00010000:00080000:13.0:1524592183.959589:0:20038:0:(ldlm_lib.c:2414:target_queue_recovery_request()) Next recovery transno: 2937774724128, current: 2937774019077, replaying 00010000:00080000:13.0:1524592183.959720:0:20038:0:(ldlm_lib.c:2414:target_queue_recovery_request()) Next recovery transno: 2937774724128, current: 2937774726636, replaying 00000001:00100000:5.0:1524592343.671937:0:19948:0:(tgt_lastrcvd.c:872:tgt_cb_new_client()) snx11205-MDT0001: committing for initial connect of 061cd4a0-cd22-5048-946a-d8f4ec53917a 00010000:00080000:8.0:1524592343.672696:0:20105:0:(ldlm_lib.c:2187:target_recovery_thread()) @@@ processing final ping from 54@gni: req@ffff880f70d39b00 x1598582492487088/t0(0) o400->061cd4a0-cd22-5048-946a-d8f4ec53917a@54@gni:-1/-1 lens 224/0 e 5 to 0 dl 1524592362 ref 1 fl Complete:H/c0/ffffffff rc 0/-1 00010000:00080000:8.0:1524592343.672703:0:20105:0:(ldlm_lib.c:2492:target_committed_to_req()) 061cd4a0-cd22-5048-946a-d8f4ec53917a last_committed 2937774968811, transno 0, xid 1598582492487088 00000100:00100000:2.0:1524592343.721133:0:22046:0:(service.c:2103:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc mdt01_005:061cd4a0-cd22-5048-946a-d8f4ec53917a+7:15972:x1598582492486880:12345-54@gni:101 00000004:00080000:2.0:1524592343.721151:0:22046:0:(mdt_recovery.c:198:mdt_req_from_lrd()) @@@ restoring transno 2937774968811/status 0 req@ffff880f6ef32a00 x1598582492486880/t0(0) o101->061cd4a0-cd22-5048-946a-d8f4ec53917a@54@gni:-1/-1 lens 656/3512 e 0 to 0 dl 1524592398 ref 1 fl Interpret:/2/0 rc 0/0 00000004:00100000:2.0:1524592343.721171:0:22046:0:(mdt_recovery.c:211:mdt_req_from_lrd()) @@@ restoring transno 2937774968811/status 0 req@ffff880f6ef32a00 x1598582492486880/t2937774968811(0) o101->061cd4a0-cd22-5048-946a-d8f4ec53917a@54@gni:-1/-1 lens 656/3512 e 0 to 0 dl 1524592398 ref 1 fl Interpret:/2/0 rc 0/0 00000004:00080000:2.0:1524592343.731829:0:22046:0:(mdt_open.c:674:mdt_mfd_set_mode()) [0x340148dee:0x127df:0x0] Change mfd mode 0 -> 01102. 00010000:00080000:2.0:1524592343.731837:0:22046:0:(ldlm_lib.c:2492:target_committed_to_req()) 061cd4a0-cd22-5048-946a-d8f4ec53917a last_committed 2937774968811, transno 2937774968811, xid 1598582492486880 00000100:00100000:2.0:1524592343.731846:0:22046:0:(service.c:2153:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc mdt01_005:061cd4a0-cd22-5048-946a-d8f4ec53917a+7:15972:x1598582492486880:12345-54@gni:101 Request procesed in 10714us (49134us total) trans 2937774968811 rc 301/301