00000100:00000200:12.0F:1547251522.042705:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441296, portal 4 00000100:00000200:2.0F:1547251522.042708:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441280, portal 4 00000100:00000200:12.0:1547251522.042709:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441296, offset 0 00000400:00000200:12.0:1547251522.042712:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000100:00000200:2.0:1547251522.042712:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441280, offset 0 00000400:00000200:12.0:1547251522.042716:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:2.0:1547251522.042716:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:12.0:1547251522.042718:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22953] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251522.042719:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:2.0:1547251522.042721:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22953] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251522.042722:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22952] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22953] 00000400:00000200:12.0:1547251522.042724:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251522.042724:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22952] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22953] 00000400:00000200:12.0:1547251522.042726:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251522.042727:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251522.042727:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:2.0:1547251522.042728:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251522.042729:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9714, 9715] 00000400:00000200:2.0:1547251522.042730:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251522.042731:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:2.0:1547251522.042731:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9714, 9715] 00000100:00000200:6.0F:1547251522.042733:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441264, portal 4 00000400:00000200:2.0:1547251522.042733:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:12.0:1547251522.042736:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251522.042739:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000100:00000200:6.0:1547251522.042742:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441264, offset 0 00000800:00000200:12.0:1547251522.042743:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000400:00000200:2.0:1547251522.042743:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251522.042744:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:12.0:1547251522.042745:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:2.0:1547251522.042745:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:6.0:1547251522.042746:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:12.0:1547251522.042748:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000400:00000200:6.0:1547251522.042749:0:12674:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:2.0:1547251522.042749:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251522.042750:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000400:00000200:6.0:1547251522.042751:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22954] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:2.0:1547251522.042751:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:2.0:1547251522.042754:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000400:00000200:6.0:1547251522.042755:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22953] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22954] 00000400:00000200:6.0:1547251522.042759:0:12674:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:6.0:1547251522.042761:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251522.042763:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251522.042765:0:12674:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9715, 9716] 00000800:00000200:7.2F:1547251522.042767:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000400:00000200:6.0:1547251522.042768:0:12674:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:6.0:1547251522.042774:0:12674:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:6.0:1547251522.042778:0:12674:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:5.2F:1547251522.042778:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:6.0:1547251522.042782:0:12674:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:6.0:1547251522.042784:0:12674:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (22)++ 00000800:00000200:6.0:1547251522.042785:0:12674:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (23)++ 00000800:00000200:6.0:1547251522.042789:0:12674:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (24)-- 00000800:00000200:9.0F:1547251522.042825:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:9.0:1547251522.042829:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000100:00000200:9.0:1547251522.042832:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b2dd100 x1622384012441296/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251522.042838:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038440 00000400:00000200:9.0:1547251522.042839:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0F:1547251522.042841:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000400:00000200:9.0:1547251522.042842:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251522.042844:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000800:00000200:9.0:1547251522.042845:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:9.0:1547251522.042846:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:9.0:1547251522.042847:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000100:00000200:10.0:1547251522.042848:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f7200 x1622384012441280/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251522.042849:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c435700 x1622384012441264/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251522.042853:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a880 00000400:00000200:9.0:1547251522.042853:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990990 00000400:00000200:9.0:1547251522.042854:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251522.042855:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251522.042856:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251522.042857:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:9.0:1547251522.042858:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:9.0:1547251522.042859:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251522.042860:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:9.0:1547251522.042862:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:5.2:1547251522.042925:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251522.042930:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:9.0:1547251522.042945:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251522.042946:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:9.0:1547251522.042948:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251522.042949:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000400:00000200:9.0:1547251522.042951:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251522.042953:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:9.0:1547251522.042955:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c036c0 00000400:00000200:10.0:1547251522.042956:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c036d0 00000400:00000200:9.0:1547251522.042958:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd61a9 [1] + 192 00000400:00000200:10.0:1547251522.042963:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6199 [1] + 192 00000100:00000200:9.0:1547251522.042968:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f7200 x1622384012441280/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251522.042978:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251522.042980:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b2dd100 x1622384012441296/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:9.0:1547251522.042980:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:9.0:1547251522.042982:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:9.0:1547251522.042983:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:9.0:1547251522.042984:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:9.0:1547251522.042986:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:9.0:1547251522.042987:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251522.042990:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251522.042990:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251522.042993:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (23)++ 00000100:00000200:2.0:1547251522.042993:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f7200 x1622384012441280/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251522.042994:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (24)-- 00000400:00000200:9.0:1547251522.042994:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c036b0 00000800:00000200:10.0:1547251522.042996:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251522.042997:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000100:00000200:2.0:1547251522.042998:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f7200 x1622384012441280/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251522.043002:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06abb0 00000400:00000200:9.0:1547251522.043005:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd61b9 [1] + 192 00010000:00080000:2.0:1547251522.043006:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f7200 x1622384012441280/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:12.0:1547251522.043008:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b2dd100 x1622384012441296/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251522.043014:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b2dd100 x1622384012441296/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251522.043018:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038880 00000100:00000200:9.0:1547251522.043020:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c435700 x1622384012441264/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:12.0:1547251522.043022:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b17b2dd100 x1622384012441296/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:9.0:1547251522.043027:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:9.0:1547251522.043029:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (21)++ 00000800:00000200:9.0:1547251522.043030:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (22)-- 00000800:00000200:9.0:1547251522.043031:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:6.0:1547251522.043038:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c435700 x1622384012441264/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251522.043043:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c435700 x1622384012441264/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251522.043047:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00010000:00080000:6.0:1547251522.043051:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ae5c435700 x1622384012441264/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0F:1547251522.506680:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441312, portal 4 00000100:00000200:8.0:1547251522.506684:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441312, offset 0 00000400:00000200:8.0:1547251522.506687:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000100:00000200:13.0F:1547251522.506688:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441328, portal 4 00000400:00000200:8.0:1547251522.506690:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251522.506692:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22954] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:13.0:1547251522.506694:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441328, offset 0 00000400:00000200:8.0:1547251522.506695:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22954] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22954] 00000400:00000200:13.0:1547251522.506697:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251522.506698:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251522.506699:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251522.506700:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251522.506700:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251522.506702:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22955] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251522.506702:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9708, 9708] 00000400:00000200:8.0:1547251522.506704:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:13.0:1547251522.506706:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22954] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22955] 00000400:00000200:13.0:1547251522.506708:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:8.0:1547251522.506709:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:13.0:1547251522.506710:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251522.506711:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251522.506711:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:13.0:1547251522.506713:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9708, 9709] 00000400:00000200:13.0:1547251522.506714:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:8.0:1547251522.506715:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251522.506716:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:8.0:1547251522.506717:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000100:00000200:2.0:1547251522.506718:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441344, portal 4 00000400:00000200:13.0:1547251522.506719:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251522.506720:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000100:00000200:2.0:1547251522.506721:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441344, offset 0 00000800:00000200:13.0:1547251522.506722:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:2.0:1547251522.506724:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:13.0:1547251522.506725:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251522.506727:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000400:00000200:2.0:1547251522.506727:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:13.0:1547251522.506729:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:2.0:1547251522.506729:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22955] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:13.0:1547251522.506732:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:2.0:1547251522.506732:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22955] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22955] 00000400:00000200:2.0:1547251522.506735:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:12.2F:1547251522.506737:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000400:00000200:2.0:1547251522.506737:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251522.506739:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251522.506740:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9709, 9709] 00000400:00000200:2.0:1547251522.506742:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:2.0:1547251522.506746:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:0.2F:1547251522.506747:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:2.0:1547251522.506749:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:2.0:1547251522.506752:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251522.506754:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (22)++ 00000800:00000200:2.0:1547251522.506755:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00000800:00000200:2.0:1547251522.506758:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (24)-- 00000800:00000200:4.0F:1547251522.506793:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:4.0:1547251522.506800:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000100:00000200:4.0:1547251522.506803:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824756600 x1622384012441312/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:7.0F:1547251522.506808:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000400:00000200:4.0:1547251522.506809:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688e58 00000400:00000200:4.0:1547251522.506810:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:7.0:1547251522.506812:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:4.0:1547251522.506812:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251522.506814:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000100:00000200:7.0:1547251522.506816:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af2400 x1622384012441328/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251522.506816:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251522.506817:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000100:00000200:4.0:1547251522.506819:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328db600 x1622384012441344/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251522.506821:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792bd48 00000400:00000200:7.0:1547251522.506822:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251522.506823:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a6e8 00000400:00000200:4.0:1547251522.506824:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251522.506825:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251522.506826:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:7.0:1547251522.506828:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251522.506829:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:7.0:1547251522.506830:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:4.0:1547251522.506831:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:0.2:1547251522.506900:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251522.506905:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:7.0:1547251522.506912:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:7.0:1547251522.506915:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000800:00000200:0.2:1547251522.506917:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:4.0:1547251522.506918:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000400:00000200:7.0:1547251522.506919:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251522.506920:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251522.506922:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c036f0 00000400:00000200:4.0:1547251522.506924:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:7.0:1547251522.506925:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe2e55 [1] + 192 00000400:00000200:4.0:1547251522.506927:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c036e0 00000100:00000200:7.0:1547251522.506929:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af2400 x1622384012441328/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251522.506948:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:7.0:1547251522.506950:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000400:00000200:4.0:1547251522.506950:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe2e45 [1] + 192 00000800:00000200:7.0:1547251522.506952:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:7.0:1547251522.506953:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:7.0:1547251522.506954:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:4.0:1547251522.506954:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824756600 x1622384012441312/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:7.0:1547251522.506956:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:7.0:1547251522.506958:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251522.506961:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:7.0:1547251522.506963:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03700 00000400:00000200:4.0:1547251522.506964:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251522.506966:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000100:00000200:13.0:1547251522.506967:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af2400 x1622384012441328/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251522.506968:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251522.506969:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251522.506970:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:13.0:1547251522.506972:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af2400 x1622384012441328/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251522.506972:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:13.0:1547251522.506976:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b2a8 00000400:00000200:7.0:1547251522.506979:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe2e65 [1] + 192 00010000:00080000:13.0:1547251522.506980:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af2400 x1622384012441328/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251522.506983:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824756600 x1622384012441312/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251522.506988:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824756600 x1622384012441312/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251522.506992:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688330 00000100:00000200:7.0:1547251522.506995:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328db600 x1622384012441344/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251522.506996:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824756600 x1622384012441312/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:7.0:1547251522.507002:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:7.0:1547251522.507004:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:7.0:1547251522.507005:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:7.0:1547251522.507006:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000100:00000200:2.0:1547251522.507012:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328db600 x1622384012441344/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251522.507017:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328db600 x1622384012441344/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251522.507021:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06abb0 00010000:00080000:2.0:1547251522.507025:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328db600 x1622384012441344/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0F:1547251522.906679:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441360, portal 4 00000100:00000200:11.0:1547251522.906683:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441360, offset 0 00000400:00000200:11.0:1547251522.906685:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:11.0:1547251522.906688:0:12681:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:11.0:1547251522.906691:0:12681:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03710 00000400:00000200:11.0:1547251522.906693:0:12681:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd3b89 [1] + 0 00000400:00000200:11.0:1547251522.906695:0:12681:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d688770 00000100:00000200:11.0:1547251522.906697:0:12681:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:11.0:1547251522.906706:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688770 00000400:00000200:11.0:1547251522.906707:0:12681:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251522.906709:0:12681:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07d700 x1622384012441360/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251522.906714:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bdd0 00000400:00000200:11.0:1547251522.906714:0:12681:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251522.906716:0:12681:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251522.906737:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441360 00000100:00000200:7.0:1547251522.906741:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441376, portal 4 00010000:00000200:8.0:1547251522.906747:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3dfc50 x1622384012441360/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:583/0 lens 224/368 e 0 to 0 dl 1547251528 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:7.0:1547251522.906747:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441376, offset 0 00000400:00000200:7.0:1547251522.906750:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251522.906753:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:8.0:1547251522.906754:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012441360, offset 192 00000400:00000200:7.0:1547251522.906756:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03720 00000400:00000200:8.0:1547251522.906757:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251522.906758:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1545 [1] + 0 00000400:00000200:8.0:1547251522.906760:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251522.906761:0:12675:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad110 00000400:00000200:8.0:1547251522.906763:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03710 00000100:00000200:7.0:1547251522.906763:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251522.906765:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae611 [1] + 192 00000100:00000200:8.0:1547251522.906772:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07d700 x1622384012441360/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251522.906772:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad110 00000400:00000200:7.0:1547251522.906773:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251522.906776:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae6aa84800 x1622384012441376/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251522.906781:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251522.906781:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea4c8 00000400:00000200:7.0:1547251522.906781:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251522.906782:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:8.0:1547251522.906783:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688330 00000400:00000200:8.0:1547251522.906784:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251522.906785:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:11.0:1547251522.906789:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07d700 x1622384012441360/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251522.906790:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:11.0:1547251522.906793:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07d700 x1622384012441360/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251522.906795:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441376 00000400:00000200:11.0:1547251522.906798:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0baa0 00010000:00080000:11.0:1547251522.906801:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b17b07d700 x1622384012441360/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/0 rc 0/0 00010000:00000200:4.0:1547251522.906808:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11d450 x1622384012441376/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:583/0 lens 224/368 e 0 to 0 dl 1547251528 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251522.906818:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012441376, offset 192 00000400:00000200:4.0:1547251522.906822:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251522.906826:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251522.906829:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03720 00000400:00000200:4.0:1547251522.906831:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae621 [1] + 192 00000100:00000200:4.0:1547251522.906835:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae6aa84800 x1622384012441376/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251522.906844:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251522.906846:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b7f8 00000400:00000200:4.0:1547251522.906847:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251522.906848:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251522.906853:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae6aa84800 x1622384012441376/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251522.906854:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251522.906858:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae6aa84800 x1622384012441376/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251522.906862:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00010000:00080000:7.0:1547251522.906865:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae6aa84800 x1622384012441376/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251522.922706:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441392, portal 4 00000100:00000200:6.0:1547251522.922710:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441392, offset 0 00000400:00000200:6.0:1547251522.922714:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251522.922717:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251522.922720:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03730 00000400:00000200:6.0:1547251522.922722:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1575 [1] + 0 00000400:00000200:6.0:1547251522.922724:0:12676:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073addd0 00000100:00000200:6.0:1547251522.922726:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251522.922736:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073addd0 00000400:00000200:6.0:1547251522.922737:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251522.922740:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae6aa84800 x1622384012441392/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251522.922744:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990bb0 00000400:00000200:6.0:1547251522.922745:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251522.922746:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251522.922757:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441392 00010000:00000200:4.0:1547251522.922770:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11d050 x1622384012441392/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:583/0 lens 224/368 e 0 to 0 dl 1547251528 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251522.922778:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012441392, offset 192 00000400:00000200:4.0:1547251522.922782:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251522.922785:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251522.922789:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03730 00000400:00000200:4.0:1547251522.922791:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae631 [1] + 192 00000100:00000200:4.0:1547251522.922794:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae6aa84800 x1622384012441392/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251522.922803:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251522.922806:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040baa0 00000400:00000200:4.0:1547251522.922806:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251522.922807:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251522.922812:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae6aa84800 x1622384012441392/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251522.922813:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251522.922817:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae6aa84800 x1622384012441392/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251522.922821:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00010000:00080000:6.0:1547251522.922824:0:12676:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae6aa84800 x1622384012441392/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251529 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251524.938677:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441408, portal 4 00000100:00000200:11.0:1547251524.938680:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441408, offset 0 00000400:00000200:11.0:1547251524.938683:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:11.0:1547251524.938686:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:11.0:1547251524.938688:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22956] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:11.0:1547251524.938691:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22955] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22956] 00000400:00000200:11.0:1547251524.938693:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:11.0:1547251524.938695:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251524.938696:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251524.938697:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9710, 9710] 00000400:00000200:11.0:1547251524.938699:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:11.0:1547251524.938705:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000100:00000200:2.0:1547251524.938707:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441440, portal 4 00000800:00000200:11.0:1547251524.938708:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:11.0:1547251524.938711:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000100:00000200:2.0:1547251524.938711:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441440, offset 0 00000800:00000200:11.0:1547251524.938712:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:11.0:1547251524.938714:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:2.0:1547251524.938715:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:11.0:1547251524.938717:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:2.0:1547251524.938718:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:2.0:1547251524.938720:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22956] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251524.938724:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22956] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22956] 00000400:00000200:2.0:1547251524.938726:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:2.0:1547251524.938727:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251524.938729:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251524.938730:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9710, 9711] 00000100:00000200:12.0:1547251524.938731:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441424, portal 4 00000800:00000200:9.2F:1547251524.938732:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:2.0:1547251524.938732:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000100:00000200:12.0:1547251524.938735:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441424, offset 0 00000400:00000200:2.0:1547251524.938737:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251524.938739:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:12.0:1547251524.938740:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:12.0:1547251524.938743:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000800:00000200:2.0:1547251524.938743:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000400:00000200:12.0:1547251524.938745:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22957] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:2.0:1547251524.938745:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:2.0:1547251524.938746:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:12.0:1547251524.938749:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22956] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22957] 00000800:00000200:2.0:1547251524.938749:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:12.0:1547251524.938751:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:12.0:1547251524.938753:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251524.938755:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251524.938756:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9711, 9711] 00000400:00000200:12.0:1547251524.938758:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251524.938763:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251524.938765:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:12.0:1547251524.938768:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251524.938770:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251524.938771:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:12.0:1547251524.938774:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (24)-- 00000800:00000200:12.0:1547251524.938782:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:12.0:1547251524.938791:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (24)-- 00000100:00000200:12.0:1547251524.938795:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07d400 x1622384012441408/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251531 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:7.2:1547251524.938800:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:12.0:1547251524.938802:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bdd0 00000800:00000200:13.0:1547251524.938804:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)++ 00000400:00000200:12.0:1547251524.938804:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:13.0:1547251524.938807:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (24)-- 00000400:00000200:12.0:1547251524.938807:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000100:00000200:13.0:1547251524.938810:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a1200 x1622384012441424/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251531 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251524.938810:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:12.0:1547251524.938812:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251524.938814:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000400:00000200:13.0:1547251524.938816:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038198 00000100:00000200:12.0:1547251524.938817:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f7500 x1622384012441440/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251531 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251524.938818:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251524.938821:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251524.938823:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:12.0:1547251524.938823:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a5d8 00000400:00000200:12.0:1547251524.938824:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:13.0:1547251524.938827:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000400:00000200:12.0:1547251524.938827:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251524.938828:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251524.938830:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:7.2:1547251524.938890:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:9.2:1547251524.938899:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:13.0:1547251524.938907:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:13.0:1547251524.938911:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.93@o2ib10 00000800:00000200:12.0:1547251524.938914:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:13.0:1547251524.938916:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:7.2:1547251524.938916:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:12.0:1547251524.938917:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.93@o2ib10 00000400:00000200:13.0:1547251524.938921:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03760 00000400:00000200:12.0:1547251524.938922:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251524.938925:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f47d [1] + 192 00000400:00000200:12.0:1547251524.938926:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03740 00000400:00000200:12.0:1547251524.938929:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f46d [1] + 192 00000100:00000200:13.0:1547251524.938934:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f7500 x1622384012441440/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251531 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251524.938954:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:12.0:1547251524.938956:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07d400 x1622384012441408/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251531 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251524.938957:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:13.0:1547251524.938959:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:13.0:1547251524.938960:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:13.0:1547251524.938962:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:13.0:1547251524.938963:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:13.0:1547251524.938965:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000400:00000200:12.0:1547251524.938967:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251524.938968:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251524.938969:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (21)++ 00000100:00000200:2.0:1547251524.938969:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f7500 x1622384012441440/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251531 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251524.938970:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:13.0:1547251524.938972:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251524.938972:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:12.0:1547251524.938973:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000100:00000200:2.0:1547251524.938975:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f7500 x1622384012441440/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251531 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251524.938976:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03750 00000400:00000200:2.0:1547251524.938979:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06abb0 00000100:00000200:11.0:1547251524.938982:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07d400 x1622384012441408/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251531 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:2.0:1547251524.938982:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f7500 x1622384012441440/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251531 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251524.938987:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07d400 x1622384012441408/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251531 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251524.938992:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0baa0 00000400:00000200:13.0:1547251524.938995:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f48d [1] + 192 00010000:00080000:11.0:1547251524.938995:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b17b07d400 x1622384012441408/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251531 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:13.0:1547251524.938998:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a1200 x1622384012441424/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251531 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251524.939006:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251524.939007:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (21)++ 00000800:00000200:13.0:1547251524.939010:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (22)-- 00000800:00000200:13.0:1547251524.939011:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:12.0:1547251524.939015:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a1200 x1622384012441424/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251531 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251524.939022:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a1200 x1622384012441424/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251531 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251524.939028:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038880 00010000:00080000:12.0:1547251524.939034:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a1200 x1622384012441424/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251531 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:0.0F:1547251527.050673:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441456, portal 4 00000100:00000200:7.0:1547251527.050677:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441472, portal 4 00000100:00000200:0.0:1547251527.050677:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441456, offset 0 00000100:00000200:7.0:1547251527.050680:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441472, offset 0 00000400:00000200:0.0:1547251527.050680:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:0.0:1547251527.050682:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:7.0:1547251527.050684:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:0.0:1547251527.050685:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22957] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:7.0:1547251527.050686:0:12677:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:0.0:1547251527.050688:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22957] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22957] 00000400:00000200:7.0:1547251527.050689:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22957] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251527.050690:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:7.0:1547251527.050692:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22957] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22957] 00000400:00000200:0.0:1547251527.050692:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251527.050693:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251527.050695:0:12677:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251527.050695:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9716, 9716] 00000400:00000200:7.0:1547251527.050696:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251527.050696:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:7.0:1547251527.050698:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251527.050699:0:12677:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9716, 9717] 00000400:00000200:7.0:1547251527.050701:0:12677:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:0.0:1547251527.050701:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251527.050704:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:7.0:1547251527.050707:0:12677:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251527.050707:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251527.050708:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:0.0:1547251527.050709:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:7.0:1547251527.050710:0:12677:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:0.0:1547251527.050712:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000800:00000200:7.0:1547251527.050713:0:12677:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:7.0:1547251527.050714:0:12677:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:7.0:1547251527.050715:0:12677:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:7.0:1547251527.050718:0:12677:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000800:00000200:7.2:1547251527.050727:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000100:00000200:15.0F:1547251527.050732:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441488, portal 4 00000800:00000200:5.2:1547251527.050735:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000100:00000200:15.0:1547251527.050738:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441488, offset 0 00000400:00000200:15.0:1547251527.050741:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:15.0:1547251527.050744:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:15.0:1547251527.050747:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22958] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251527.050750:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22958] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22958] 00000400:00000200:15.0:1547251527.050752:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:15.0:1547251527.050754:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251527.050755:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251527.050756:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9717, 9717] 00000400:00000200:15.0:1547251527.050758:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:15.0:1547251527.050763:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251527.050766:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:15.0:1547251527.050769:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251527.050770:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (22)++ 00000800:00000200:15.0:1547251527.050771:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (23)++ 00000800:00000200:15.0:1547251527.050774:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (24)-- 00000800:00000200:10.0:1547251527.050780:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251527.050783:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000100:00000200:10.0:1547251527.050787:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df5d00 x1622384012441472/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251527.050792:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea4c8 00000400:00000200:10.0:1547251527.050793:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251527.050795:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:9.0:1547251527.050795:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (23)++ 00000800:00000200:10.0:1547251527.050798:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:9.0:1547251527.050798:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (24)-- 00000800:00000200:10.0:1547251527.050799:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:10.0:1547251527.050801:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (23)++ 00000800:00000200:10.0:1547251527.050802:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (24)-- 00000100:00000200:9.0:1547251527.050802:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c6600 x1622384012441456/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:10.0:1547251527.050804:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af7500 x1622384012441488/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251527.050807:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246653b8 00000400:00000200:9.0:1547251527.050808:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251527.050809:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1b28 00000400:00000200:10.0:1547251527.050811:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251527.050811:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251527.050813:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:9.0:1547251527.050813:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (23)-- 00000800:00000200:10.0:1547251527.050814:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:9.0:1547251527.050816:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:5.2:1547251527.050885:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251527.050891:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:10.0:1547251527.050897:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251527.050900:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251527.050903:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:9.0:1547251527.050904:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000400:00000200:10.0:1547251527.050907:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03770 00000800:00000200:9.0:1547251527.050907:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251527.050910:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6219 [1] + 192 00000400:00000200:9.0:1547251527.050910:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000100:00000200:10.0:1547251527.050914:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c6600 x1622384012441456/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251527.050914:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03780 00000400:00000200:10.0:1547251527.050924:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251527.050926:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000400:00000200:9.0:1547251527.050926:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6229 [1] + 192 00000800:00000200:10.0:1547251527.050927:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:10.0:1547251527.050928:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251527.050930:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000100:00000200:9.0:1547251527.050930:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df5d00 x1622384012441472/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251527.050931:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251527.050934:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251527.050943:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03790 00000400:00000200:9.0:1547251527.050947:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:9.0:1547251527.050949:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000100:00000200:0.0:1547251527.050949:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c6600 x1622384012441456/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:9.0:1547251527.050950:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:9.0:1547251527.050952:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:9.0:1547251527.050953:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:9.0:1547251527.050954:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000100:00000200:0.0:1547251527.050955:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c6600 x1622384012441456/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251527.050959:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00000400:00000200:10.0:1547251527.050961:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6239 [1] + 192 00010000:00080000:0.0:1547251527.050962:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c6600 x1622384012441456/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251527.050965:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df5d00 x1622384012441472/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251527.050970:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df5d00 x1622384012441472/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251527.050975:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00000100:00000200:10.0:1547251527.050977:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af7500 x1622384012441488/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:7.0:1547251527.050978:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a765df5d00 x1622384012441472/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251527.050984:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251527.050986:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251527.050987:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251527.050988:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:15.0:1547251527.050995:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af7500 x1622384012441488/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:15.0:1547251527.051000:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af7500 x1622384012441488/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251527.051005:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00010000:00080000:15.0:1547251527.051008:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af7500 x1622384012441488/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0F:1547251527.514708:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441520, portal 4 00000100:00000200:9.0:1547251527.514712:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441536, portal 4 00000100:00000200:3.0:1547251527.514712:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441520, offset 0 00000100:00000200:9.0:1547251527.514715:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441536, offset 0 00000400:00000200:3.0:1547251527.514715:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251527.514718:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:3.0:1547251527.514718:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:3.0:1547251527.514720:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22959] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:15.0:1547251527.514722:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441504, portal 4 00000400:00000200:9.0:1547251527.514722:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251527.514724:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22959] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251527.514724:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22958] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22959] 00000100:00000200:15.0:1547251527.514726:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441504, offset 0 00000400:00000200:3.0:1547251527.514726:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:3.0:1547251527.514727:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251527.514728:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22959] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22959] 00000400:00000200:3.0:1547251527.514729:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251527.514730:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251527.514730:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9709, 9710] 00000400:00000200:15.0:1547251527.514731:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251527.514732:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251527.514732:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:9.0:1547251527.514733:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251527.514734:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251527.514734:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9710, 9710] 00000400:00000200:15.0:1547251527.514737:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22960] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251527.514737:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:3.0:1547251527.514737:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:15.0:1547251527.514740:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22959] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22960] 00000800:00000200:3.0:1547251527.514740:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:15.0:1547251527.514742:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251527.514743:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251527.514743:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251527.514744:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000400:00000200:15.0:1547251527.514745:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251527.514745:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:3.0:1547251527.514745:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:15.0:1547251527.514747:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251527.514748:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9710, 9711] 00000800:00000200:3.0:1547251527.514748:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:9.0:1547251527.514749:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:15.0:1547251527.514750:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:9.0:1547251527.514750:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:9.0:1547251527.514751:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:9.0:1547251527.514754:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000400:00000200:15.0:1547251527.514755:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251527.514758:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:15.0:1547251527.514761:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251527.514762:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:15.0:1547251527.514763:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:0.2:1547251527.514763:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:15.0:1547251527.514766:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:12.2:1547251527.514774:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:12.2:1547251527.514783:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251527.514816:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251527.514819:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:4.0:1547251527.514823:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c6300 x1622384012441520/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251527.514828:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245555d8 00000400:00000200:4.0:1547251527.514829:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:7.0:1547251527.514832:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000400:00000200:4.0:1547251527.514832:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:7.0:1547251527.514835:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251527.514835:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251527.514837:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251527.514838:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000100:00000200:7.0:1547251527.514839:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf0600 x1622384012441536/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251527.514841:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b2d8600 x1622384012441504/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251527.514844:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ead48 00000400:00000200:4.0:1547251527.514845:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1110 00000400:00000200:7.0:1547251527.514846:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251527.514846:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251527.514848:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251527.514848:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251527.514849:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:7.0:1547251527.514851:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251527.514852:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:7.0:1547251527.514853:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:4.0:1547251527.514854:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:0.2:1547251527.514920:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251527.514926:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:12.2:1547251527.514932:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:7.0:1547251527.514935:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:7.0:1547251527.514946:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251527.514947:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000400:00000200:7.0:1547251527.514950:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251527.514950:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.92@o2ib10 00000400:00000200:7.0:1547251527.514954:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c037a0 00000400:00000200:4.0:1547251527.514955:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:7.0:1547251527.514957:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe2f05 [1] + 192 00000400:00000200:4.0:1547251527.514959:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c037b0 00000100:00000200:7.0:1547251527.514961:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b2d8600 x1622384012441504/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251527.514972:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251527.514973:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe2ee5 [1] + 192 00000800:00000200:7.0:1547251527.514974:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:7.0:1547251527.514976:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:7.0:1547251527.514977:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:4.0:1547251527.514977:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c6300 x1622384012441520/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:7.0:1547251527.514979:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:7.0:1547251527.514981:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251527.514983:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:7.0:1547251527.514986:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c037c0 00000400:00000200:4.0:1547251527.514987:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251527.514989:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000100:00000200:15.0:1547251527.514990:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b2d8600 x1622384012441504/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251527.514991:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:4.0:1547251527.514992:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251527.514993:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:15.0:1547251527.514995:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b2d8600 x1622384012441504/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251527.514995:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:4.0:1547251527.514996:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:15.0:1547251527.515000:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00000400:00000200:7.0:1547251527.515002:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe2ef5 [1] + 192 00010000:00080000:15.0:1547251527.515004:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b17b2d8600 x1622384012441504/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251527.515006:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c6300 x1622384012441520/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251527.515012:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c6300 x1622384012441520/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251527.515016:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00000100:00000200:7.0:1547251527.515018:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf0600 x1622384012441536/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:3.0:1547251527.515019:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c6300 x1622384012441520/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:7.0:1547251527.515025:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:7.0:1547251527.515026:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000800:00000200:7.0:1547251527.515027:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:7.0:1547251527.515028:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:9.0:1547251527.515079:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf0600 x1622384012441536/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251527.515085:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf0600 x1622384012441536/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251527.515090:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eae58 00010000:00080000:9.0:1547251527.515093:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf0600 x1622384012441536/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251527.914720:0:12678:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441552, portal 4 00000100:00000200:10.0:1547251527.914723:0:12678:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441552, offset 0 00000100:00000200:6.0:1547251527.914724:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441568, portal 4 00000400:00000200:10.0:1547251527.914725:0:12678:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:6.0:1547251527.914727:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441568, offset 0 00000400:00000200:10.0:1547251527.914728:0:12678:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251527.914730:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:10.0:1547251527.914731:0:12678:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c037d0 00000400:00000200:10.0:1547251527.914733:0:12678:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd3c69 [1] + 0 00000400:00000200:6.0:1547251527.914733:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:10.0:1547251527.914735:0:12678:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ae6a9144c8 00000400:00000200:6.0:1547251527.914735:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c037e0 00000100:00000200:10.0:1547251527.914738:0:12678:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251527.914738:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe15f5 [1] + 0 00000400:00000200:6.0:1547251527.914740:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b6459906e8 00000100:00000200:6.0:1547251527.914742:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:10.0:1547251527.914746:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a9144c8 00000400:00000200:10.0:1547251527.914747:0:12678:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251527.914749:0:12678:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960ed700 x1622384012441552/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251527.914751:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459906e8 00000400:00000200:6.0:1547251527.914752:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:10.0:1547251527.914754:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914ee0 00000100:00000200:6.0:1547251527.914754:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37932700 x1622384012441568/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251527.914755:0:12678:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251527.914756:0:12678:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251527.914759:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990bb0 00000400:00000200:6.0:1547251527.914759:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251527.914761:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251527.914776:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441552 00010000:00000200:8.0:1547251527.914785:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3df850 x1622384012441552/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:588/0 lens 224/368 e 0 to 0 dl 1547251533 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0F:1547251527.914785:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441568 00000100:00000200:8.0:1547251527.914791:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012441552, offset 192 00000400:00000200:8.0:1547251527.914794:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00010000:00000200:5.0:1547251527.914795:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11cc50 x1622384012441568/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:588/0 lens 224/368 e 0 to 0 dl 1547251533 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:8.0:1547251527.914797:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251527.914800:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c037d0 00000400:00000200:8.0:1547251527.914802:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae671 [1] + 192 00000100:00000200:5.0:1547251527.914802:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012441568, offset 192 00000400:00000200:5.0:1547251527.914804:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:8.0:1547251527.914805:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960ed700 x1622384012441552/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251527.914807:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251527.914811:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c037e0 00000400:00000200:8.0:1547251527.914814:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251527.914815:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae681 [1] + 192 00000400:00000200:8.0:1547251527.914816:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688550 00000400:00000200:8.0:1547251527.914817:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251527.914818:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251527.914818:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37932700 x1622384012441568/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251527.914823:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:5.0:1547251527.914828:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251527.914830:0:12678:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960ed700 x1622384012441552/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251527.914830:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad220 00000400:00000200:5.0:1547251527.914831:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251527.914832:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:10.0:1547251527.914834:0:12678:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960ed700 x1622384012441552/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251527.914837:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:10.0:1547251527.914838:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914550 00000100:00000200:6.0:1547251527.914840:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37932700 x1622384012441568/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:10.0:1547251527.914841:0:12678:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba960ed700 x1622384012441552/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251527.914845:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37932700 x1622384012441568/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251527.914853:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00010000:00080000:6.0:1547251527.914856:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37932700 x1622384012441568/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251527.930685:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441584, portal 4 00000100:00000200:7.0:1547251527.930688:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441584, offset 0 00000400:00000200:7.0:1547251527.930690:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251527.930693:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251527.930696:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c037f0 00000400:00000200:7.0:1547251527.930699:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1625 [1] + 0 00000400:00000200:7.0:1547251527.930701:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990d48 00000100:00000200:7.0:1547251527.930703:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:7.0:1547251527.930711:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990d48 00000400:00000200:7.0:1547251527.930712:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251527.930714:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df4200 x1622384012441584/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251527.930719:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:7.0:1547251527.930719:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251527.930720:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251527.930741:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441584 00010000:00000200:5.0:1547251527.930750:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11c850 x1622384012441584/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:588/0 lens 224/368 e 0 to 0 dl 1547251533 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251527.930757:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012441584, offset 192 00000400:00000200:5.0:1547251527.930759:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251527.930762:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251527.930765:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c037f0 00000400:00000200:5.0:1547251527.930768:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae691 [1] + 192 00000100:00000200:5.0:1547251527.930771:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df4200 x1622384012441584/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251527.930780:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251527.930782:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073add48 00000400:00000200:5.0:1547251527.930783:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251527.930784:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251527.930788:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df4200 x1622384012441584/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251527.930789:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251527.930793:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df4200 x1622384012441584/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251527.930797:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00010000:00080000:7.0:1547251527.930801:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df4200 x1622384012441584/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251534 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:0.2:1547251529.565368:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:12.2:1547251529.565377:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:4.0:1547251529.565422:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251529.565426:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251529.565429:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251529.565433:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b101e0 00000800:00000200:7.0:1547251529.565437:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000400:00000200:4.0:1547251529.565437:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 55104 00000100:00000200:4.0:1547251529.565440:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:7.0:1547251529.565441:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:7.0:1547251529.565444:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:7.0:1547251529.565448:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b101f0 00000400:00000200:4.0:1547251529.565450:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251529.565452:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 55328 00000800:00000200:4.0:1547251529.565454:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000100:00000200:7.0:1547251529.565455:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:4.0:1547251529.565456:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:4.0:1547251529.565459:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251529.565460:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251529.565463:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251529.565464:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:7.0:1547251529.565467:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000400:00000200:4.0:1547251529.565468:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:7.0:1547251529.565469:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:7.0:1547251529.565470:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:7.0:1547251529.565471:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:7.0:1547251529.565473:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:7.0:1547251529.565476:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:7.0:1547251529.565478:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 26 MB=0x5c38cb6b101c0 00000400:00000200:7.0:1547251529.565481:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0xd4779 [1] + 0 00000400:00000200:7.0:1547251529.565482:0:12662:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d220 00000100:00000200:7.0:1547251529.565484:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000100:00000200:4.0:1547251529.565487:0:13338:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444720 00000100:00000200:5.0:1547251529.565489:0:12979:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444704 00000400:00000200:7.0:1547251529.565492:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d220 00000400:00000200:7.0:1547251529.565493:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00010000:00000200:5.0:1547251529.565494:0:12979:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae6aa84800 x1622384011444704/t0(0) o400->cache1-MDT0000-lwp-OST0006_UUID@10.0.11.92@o2ib10:590/0 lens 224/224 e 0 to 0 dl 1547251535 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:4.0:1547251529.565494:0:13338:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae37932700 x1622384011444720/t0(0) o400->cache1-MDT0000-lwp-OST0008_UUID@10.0.11.92@o2ib10:590/0 lens 224/224 e 0 to 0 dl 1547251535 ref 1 fl Interpret:H/0/0 rc 0/0 00000800:00000200:7.0:1547251529.565496:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:7.0:1547251529.565497:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:7.0:1547251529.565498:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:7.0:1547251529.565501:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:7.0:1547251529.565502:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000100:00000200:5.0:1547251529.565503:0:12979:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444704, offset 192 00000100:00000200:4.0:1547251529.565503:0:13338:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444720, offset 192 00000400:00000200:4.0:1547251529.565506:0:13338:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000400:00000200:5.0:1547251529.565509:0:12979:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:4.0:1547251529.565511:0:13338:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00000400:00000200:4.0:1547251529.565513:0:13338:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251529.565515:0:13338:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251529.565517:0:13338:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9711, 9711] 00000400:00000200:4.0:1547251529.565519:0:13338:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251529.565526:0:13338:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.1.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:4.0:1547251529.565530:0:13338:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:5.0:1547251529.565531:0:12979:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:5.0:1547251529.565534:0:12979:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:5.0:1547251529.565535:0:12979:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:5.0:1547251529.565537:0:12979:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9711, 9712] 00000400:00000200:5.0:1547251529.565539:0:12979:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251529.565545:0:12979:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:5.0:1547251529.565549:0:12979:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:4.0:1547251529.565550:0:13338:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000100:00000200:2.0:1547251529.565550:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444672 00000800:00000200:5.0:1547251529.565552:0:12979:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:4.0:1547251529.565552:0:13338:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:5.0:1547251529.565553:0:12979:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:4.0:1547251529.565553:0:13338:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251529.565554:0:12979:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:4.0:1547251529.565556:0:13338:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00010000:00000200:2.0:1547251529.565556:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ba7349a450 x1622384011444672/t0(0) o400->04d65b15-cf9c-7ded-611d-ffe7fe3f3abf@10.0.11.92@o2ib10:590/0 lens 224/224 e 0 to 0 dl 1547251535 ref 1 fl Interpret:/0/0 rc 0/0 00000800:00000200:5.0:1547251529.565557:0:12979:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:2.0:1547251529.565563:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384011444672, offset 192 00000800:00000200:12.2:1547251529.565565:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000400:00000200:4.0:1547251529.565566:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b101d0 00000400:00000200:2.0:1547251529.565566:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000400:00000200:4.0:1547251529.565570:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 55552 00000400:00000200:2.0:1547251529.565570:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00000800:00000200:0.2:1547251529.565570:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:2.0:1547251529.565572:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:4.0:1547251529.565573:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:2.0:1547251529.565573:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251529.565575:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9712, 9712] 00000400:00000200:2.0:1547251529.565577:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:7.0:1547251529.565578:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:7.0:1547251529.565580:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000400:00000200:4.0:1547251529.565581:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251529.565582:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b5d8 00000400:00000200:2.0:1547251529.565582:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.1.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:7.0:1547251529.565583:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:2.0:1547251529.565603:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:2.0:1547251529.565606:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000100:00000200:5.0:1547251529.565607:0:13338:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444688 00000400:00000200:7.0:1547251529.565608:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:2.0:1547251529.565608:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (22)++ 00000800:00000200:2.0:1547251529.565609:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00010000:00000200:5.0:1547251529.565611:0:13338:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444dd00 x1622384011444688/t0(0) o400->cache1-MDT0000-lwp-OST0007_UUID@10.0.11.92@o2ib10:590/0 lens 224/224 e 0 to 0 dl 1547251535 ref 1 fl Interpret:H/0/0 rc 0/0 00000800:00000200:2.0:1547251529.565612:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (24)-- 00000100:00000200:2.0:1547251529.565616:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:4.0:1547251529.565617:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:7.0:1547251529.565618:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000100:00000200:5.0:1547251529.565619:0:13338:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444688, offset 192 00000800:00000200:4.0:1547251529.565619:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:7.0:1547251529.565620:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000400:00000200:5.0:1547251529.565622:0:13338:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:4.0:1547251529.565622:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:7.0:1547251529.565623:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:7.0:1547251529.565624:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad6e8 00000400:00000200:7.0:1547251529.565625:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251529.565625:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000400:00000200:5.0:1547251529.565626:0:13338:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:7.0:1547251529.565627:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251529.565627:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000400:00000200:5.0:1547251529.565629:0:13338:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:5.0:1547251529.565630:0:13338:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251529.565630:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06abb0 00000400:00000200:5.0:1547251529.565632:0:13338:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9712, 9713] 00000400:00000200:4.0:1547251529.565632:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251529.565635:0:13338:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251529.565635:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:5.0:1547251529.565642:0:13338:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:7.0:1547251529.565645:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251529.565645:0:13338:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:4.0:1547251529.565646:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:7.0:1547251529.565647:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:5.0:1547251529.565648:0:13338:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:4.0:1547251529.565648:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:5.0:1547251529.565650:0:13338:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:5.0:1547251529.565651:0:13338:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251529.565653:0:13338:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:0.2:1547251529.565664:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:7.0:1547251529.565676:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:7.0:1547251529.565681:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:7.0:1547251529.565682:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adb28 00000400:00000200:7.0:1547251529.565683:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251529.565685:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:7.0:1547251529.565687:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:7.0:1547251529.565688:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:9.2:1547251529.599705:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.2:1547251529.599721:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:13.0:1547251529.599762:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251529.599766:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:13.0:1547251529.599767:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251529.599769:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:13.0:1547251529.599772:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251529.599774:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251529.599778:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb35101f0 00000400:00000200:12.0:1547251529.599778:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb35101d0 00000400:00000200:12.0:1547251529.599782:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 27008 00000400:00000200:13.0:1547251529.599786:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 27232 00000100:00000200:12.0:1547251529.599789:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:13.0:1547251529.599799:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:12.0:1547251529.599799:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251529.599801:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251529.599804:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251529.599805:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:13.0:1547251529.599806:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251529.599807:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:13.0:1547251529.599808:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:13.0:1547251529.599809:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (24)-- 00000800:00000200:13.0:1547251529.599810:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:12.0:1547251529.599810:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000800:00000200:13.0:1547251529.599812:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:13.0:1547251529.599814:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251529.599814:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251529.599818:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb35101e0 00000400:00000200:13.0:1547251529.599819:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251529.599821:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 27456 00000400:00000200:13.0:1547251529.599822:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 26 MB=0x5c38cb35101c0 00000100:00000200:12.0:1547251529.599824:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:13.0:1547251529.599826:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0xd47b9 [1] + 0 00000400:00000200:12.0:1547251529.599828:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:13.0:1547251529.599829:0:12666:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52dbb0 00000800:00000200:12.0:1547251529.599829:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000100:00000200:13.0:1547251529.599830:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000800:00000200:12.0:1547251529.599831:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:12.0:1547251529.599832:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251529.599834:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:12.0:1547251529.599836:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:13.0:1547251529.599838:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52dbb0 00000400:00000200:13.0:1547251529.599839:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251529.599841:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:13.0:1547251529.599842:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:13.0:1547251529.599843:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000100:00000200:12.0:1547251529.599855:0:12987:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821584 00010000:00000200:12.0:1547251529.599862:0:12987:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97bae1af5700 x1622383954821584/t0(0) o400->cache1-MDT0000-lwp-OST000a_UUID@10.0.11.93@o2ib10:590/0 lens 224/224 e 0 to 0 dl 1547251535 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:14.0F:1547251529.599863:0:13374:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821616 00000100:00000200:13.0:1547251529.599865:0:13371:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821600 00010000:00000200:13.0:1547251529.599870:0:13371:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97bae1af1e00 x1622383954821600/t0(0) o400->cache1-MDT0000-lwp-OST0009_UUID@10.0.11.93@o2ib10:590/0 lens 224/224 e 0 to 0 dl 1547251535 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:12.0:1547251529.599870:0:12987:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821584, offset 192 00010000:00000200:14.0:1547251529.599873:0:13374:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b17b2d8600 x1622383954821616/t0(0) o400->cache1-MDT0000-lwp-OST000b_UUID@10.0.11.93@o2ib10:590/0 lens 224/224 e 0 to 0 dl 1547251535 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:12.0:1547251529.599874:0:12987:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000100:00000200:13.0:1547251529.599877:0:13371:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821600, offset 192 00000400:00000200:12.0:1547251529.599878:0:12987:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00000400:00000200:13.0:1547251529.599880:0:13371:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:12.0:1547251529.599881:0:12987:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:14.0:1547251529.599882:0:13374:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821616, offset 192 00000400:00000200:12.0:1547251529.599883:0:12987:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251529.599885:0:12987:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9711, 9712] 00000400:00000200:14.0:1547251529.599886:0:13374:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:12.0:1547251529.599887:0:12987:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251529.599894:0:12987:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.1.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000100:00000200:2.0:1547251529.599897:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821568 00000800:00000200:12.0:1547251529.599898:0:12987:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:13.0:1547251529.599900:0:13371:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:13.0:1547251529.599903:0:13371:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00010000:00000200:2.0:1547251529.599903:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2de050 x1622383954821568/t0(0) o400->b99b3dbb-ff3e-ba67-561d-3f0ab43da035@10.0.11.93@o2ib10:590/0 lens 224/224 e 0 to 0 dl 1547251535 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:13.0:1547251529.599905:0:13371:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251529.599906:0:13371:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9712, 9712] 00000400:00000200:13.0:1547251529.599908:0:13371:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000100:00000200:2.0:1547251529.599909:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954821568, offset 192 00000400:00000200:2.0:1547251529.599913:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000400:00000200:13.0:1547251529.599915:0:13371:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000400:00000200:2.0:1547251529.599917:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00000800:00000200:13.0:1547251529.599918:0:13371:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:2.0:1547251529.599919:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251529.599920:0:13374:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:2.0:1547251529.599921:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251529.599922:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9712, 9713] 00000400:00000200:14.0:1547251529.599923:0:13374:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251529.599924:0:13374:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251529.599924:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251529.599925:0:13374:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9713, 9713] 00000400:00000200:14.0:1547251529.599927:0:13374:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251529.599933:0:13374:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251529.599943:0:13374:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:13.0:1547251529.599947:0:13371:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251529.599947:0:12987:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000400:00000200:2.0:1547251529.599947:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.1.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251529.599949:0:13371:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:12.0:1547251529.599949:0:12987:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:13.0:1547251529.599950:0:13371:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251529.599950:0:12987:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:2.0:1547251529.599950:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:14.0:1547251529.599951:0:13374:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251529.599953:0:13374:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:13.0:1547251529.599953:0:13371:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251529.599953:0:12987:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:14.0:1547251529.599954:0:13374:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:2.0:1547251529.599954:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251529.599955:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:2.0:1547251529.599956:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:14.0:1547251529.599957:0:13374:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:2.0:1547251529.599959:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:2.0:1547251529.599963:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:9.2:1547251529.599966:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:7.2:1547251529.599981:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251529.599994:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:13.0:1547251529.599996:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)++ 00000800:00000200:12.0:1547251529.599996:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (24)-- 00000800:00000200:13.0:1547251529.599998:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (24)-- 00000400:00000200:12.0:1547251529.599998:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b2a8 00000400:00000200:13.0:1547251529.599999:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038880 00000400:00000200:12.0:1547251529.599999:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251529.600000:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251529.600003:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:13.0:1547251529.600004:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251529.600005:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:13.0:1547251529.600006:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251529.600007:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:13.0:1547251529.600008:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251529.600009:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:13.0:1547251529.600010:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000400:00000200:12.0:1547251529.600010:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf606e8 00000400:00000200:12.0:1547251529.600011:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251529.600012:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a4c8 00000400:00000200:13.0:1547251529.600013:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251529.600014:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:13.0:1547251529.600016:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251529.600016:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251529.600017:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:13.0:1547251529.600019:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251529.600020:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:7.2:1547251529.631403:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:5.2:1547251529.631419:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:9.0:1547251529.631458:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:9.0:1547251529.631462:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:9.0:1547251529.631465:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:9.0:1547251529.631469:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510220 00000400:00000200:9.0:1547251529.631472:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 10752 00000100:00000200:9.0:1547251529.631475:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:10.0:1547251529.631476:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251529.631479:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251529.631483:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:9.0:1547251529.631484:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:9.0:1547251529.631487:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000400:00000200:10.0:1547251529.631489:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510210 00000800:00000200:9.0:1547251529.631489:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:9.0:1547251529.631491:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:9.0:1547251529.631493:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:10.0:1547251529.631494:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 10976 00000800:00000200:9.0:1547251529.631495:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:9.0:1547251529.631497:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000100:00000200:10.0:1547251529.631498:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:9.0:1547251529.631500:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510200 00000400:00000200:10.0:1547251529.631505:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251529.631506:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 11200 00000800:00000200:10.0:1547251529.631507:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000100:00000200:9.0:1547251529.631508:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:10.0:1547251529.631510:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251529.631511:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:9.0:1547251529.631511:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251529.631513:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:9.0:1547251529.631513:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:9.0:1547251529.631514:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:9.0:1547251529.631515:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251529.631516:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000800:00000200:9.0:1547251529.631516:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:9.0:1547251529.631518:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:10.0:1547251529.631520:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251529.631524:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 26 MB=0x5c38cb35101f0 00000400:00000200:10.0:1547251529.631527:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0xd47c9 [1] + 0 00000100:00000200:9.0:1547251529.631529:0:12982:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821664 00000100:00000200:11.0:1547251529.631530:0:12983:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821648 00000400:00000200:10.0:1547251529.631530:0:12663:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d2a8 00000100:00000200:10.0:1547251529.631532:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00010000:00000200:9.0:1547251529.631535:0:12982:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b824755700 x1622383954821664/t0(0) o400->cache1-MDT0000-lwp-OST0004_UUID@10.0.11.91@o2ib10:590/0 lens 224/224 e 0 to 0 dl 1547251535 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:11.0:1547251529.631537:0:12983:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b17b078000 x1622383954821648/t0(0) o400->cache1-MDT0000-lwp-OST0003_UUID@10.0.11.91@o2ib10:590/0 lens 224/224 e 0 to 0 dl 1547251535 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:10.0:1547251529.631542:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d2a8 00000400:00000200:10.0:1547251529.631543:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251529.631544:0:12982:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821664, offset 192 00000800:00000200:10.0:1547251529.631545:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (21)++ 00000100:00000200:11.0:1547251529.631546:0:12983:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821648, offset 192 00000800:00000200:10.0:1547251529.631547:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (22)-- 00000400:00000200:9.0:1547251529.631547:0:12982:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:10.0:1547251529.631548:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000400:00000200:11.0:1547251529.631550:0:12983:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000400:00000200:9.0:1547251529.631550:0:12982:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251529.631552:0:12982:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251529.631553:0:12982:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:10.0:1547251529.631554:0:13370:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821632 00000400:00000200:9.0:1547251529.631554:0:12982:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9717, 9718] 00000400:00000200:9.0:1547251529.631556:0:12982:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00010000:00000200:10.0:1547251529.631559:0:13370:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b824753000 x1622383954821632/t0(0) o400->cache1-MDT0000-lwp-OST0005_UUID@10.0.11.91@o2ib10:590/0 lens 224/224 e 0 to 0 dl 1547251535 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:9.0:1547251529.631561:0:12982:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251529.631564:0:12982:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:11.0:1547251529.631566:0:12983:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000100:00000200:10.0:1547251529.631566:0:13370:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821632, offset 192 00000400:00000200:11.0:1547251529.631569:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251529.631569:0:13370:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:11.0:1547251529.631571:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251529.631573:0:12983:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9718, 9718] 00000400:00000200:11.0:1547251529.631575:0:12983:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251529.631582:0:12983:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.1.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251529.631601:0:12983:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:9.0:1547251529.631601:0:12982:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000400:00000200:10.0:1547251529.631602:0:13370:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:9.0:1547251529.631603:0:12982:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:9.0:1547251529.631604:0:12982:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000100:00000200:2.0:1547251529.631604:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821616 00000400:00000200:10.0:1547251529.631605:0:13370:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251529.631606:0:12982:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000400:00000200:10.0:1547251529.631607:0:13370:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251529.631608:0:13370:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9718, 9719] 00010000:00000200:2.0:1547251529.631609:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2dd450 x1622383954821616/t0(0) o400->dc7951f7-3e71-e9c4-2e5e-5f95f4837d68@10.0.11.91@o2ib10:590/0 lens 224/224 e 0 to 0 dl 1547251535 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:10.0:1547251529.631610:0:13370:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000100:00000200:2.0:1547251529.631616:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954821616, offset 192 00000400:00000200:10.0:1547251529.631617:0:13370:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:7.2:1547251529.631618:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000400:00000200:2.0:1547251529.631619:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000800:00000200:10.0:1547251529.631620:0:13370:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:11.0:1547251529.631621:0:12983:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251529.631623:0:12983:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000400:00000200:2.0:1547251529.631623:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000800:00000200:11.0:1547251529.631624:0:12983:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251529.631624:0:13370:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251529.631624:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251529.631625:0:13370:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (23)++ 00000400:00000200:2.0:1547251529.631625:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251529.631626:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:10.0:1547251529.631627:0:13370:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (24)++ 00000800:00000200:11.0:1547251529.631628:0:12983:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000400:00000200:2.0:1547251529.631628:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[6, 7], s:[9719, 9719] 00000800:00000200:10.0:1547251529.631629:0:13370:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (25)-- 00000800:00000200:9.0:1547251529.631629:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (25)-- 00000400:00000200:2.0:1547251529.631629:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:9.0:1547251529.631631:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eae58 00000400:00000200:9.0:1547251529.631632:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251529.631635:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:2.0:1547251529.631635:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.1.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:10.0:1547251529.631637:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:10.0:1547251529.631638:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000800:00000200:9.0:1547251529.631638:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (24)-- 00000800:00000200:2.0:1547251529.631638:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:9.0:1547251529.631639:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000400:00000200:10.0:1547251529.631640:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914550 00000800:00000200:5.2:1547251529.631640:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000400:00000200:10.0:1547251529.631641:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:2.0:1547251529.631641:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251529.631642:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (22)++ 00000800:00000200:2.0:1547251529.631643:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (23)++ 00000400:00000200:10.0:1547251529.631645:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:2.0:1547251529.631646:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (24)-- 00000800:00000200:10.0:1547251529.631647:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:10.0:1547251529.631649:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (23)++ 00000100:00000200:2.0:1547251529.631650:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:10.0:1547251529.631658:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (24)-- 00000400:00000200:10.0:1547251529.631659:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0baa0 00000400:00000200:10.0:1547251529.631660:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251529.631662:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251529.631664:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (23)-- 00000800:00000200:10.0:1547251529.631665:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251529.631669:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000400:00000200:10.0:1547251529.631671:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a3b8 00000400:00000200:10.0:1547251529.631671:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:9.0:1547251529.631671:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251529.631673:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251529.631675:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:12.0:1547251529.833119:0:70:0:(o2iblnd_cb.c:2044:kiblnd_close_conn_locked()) closing conn to 10.0.10.184@o2ib10 00000800:00000200:12.0:1547251529.833122:0:70:0:(o2iblnd.c:417:kiblnd_unlink_peer_locked()) peer_ni[ffff97ae6c6da580] -> 10.0.10.184@o2ib10 (2)-- 00000800:00000200:12.0:1547251529.833129:0:70:0:(o2iblnd_cb.c:3286:kiblnd_cm_callback()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:8.2F:1547251529.833708:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:13.0:1547251529.833734:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (20)++ 00000800:00000200:13.0:1547251529.833742:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (21)-- 00000800:00000200:13.0:1547251529.833743:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:13.0:1547251529.833745:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (19)-- 00000800:00000200:9.0:1547251529.833803:0:12658:0:(o2iblnd_cb.c:3533:kiblnd_connd()) conn[ffff97a65d07cc00] (18)-- 00000800:00000200:8.2:1547251529.833811:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07cc00] (17)++ 00000800:00000200:13.0:1547251529.833824:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (18)++ 00000800:00000200:13.0:1547251529.833829:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (19)-- 00000800:00000200:13.0:1547251529.833830:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (18)-- 00000800:00000200:13.0:1547251529.833832:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (17)++ 00000800:00000200:13.0:1547251529.833834:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (18)-- 00000800:00000200:13.0:1547251529.833835:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (17)-- 00000800:00000200:13.0:1547251529.833836:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (16)++ 00000800:00000200:13.0:1547251529.833840:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (17)-- 00000800:00000200:13.0:1547251529.833841:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (16)-- 00000800:00000200:13.0:1547251529.833842:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (15)++ 00000800:00000200:13.0:1547251529.833846:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (16)-- 00000800:00000200:13.0:1547251529.833847:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (15)-- 00000800:00000200:13.0:1547251529.833848:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (14)++ 00000800:00000200:13.0:1547251529.833852:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (15)-- 00000800:00000200:13.0:1547251529.833853:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (14)-- 00000800:00000200:13.0:1547251529.833854:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (13)++ 00000800:00000200:13.0:1547251529.833857:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (14)-- 00000800:00000200:13.0:1547251529.833858:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (13)-- 00000800:00000200:13.0:1547251529.833860:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (12)++ 00000800:00000200:13.0:1547251529.833863:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (13)-- 00000800:00000200:12.0:1547251529.833865:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (13)++ 00000800:00000200:13.0:1547251529.833867:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (13)-- 00000800:00000200:13.0:1547251529.833868:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (12)++ 00000800:00000200:12.0:1547251529.833868:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (12)-- 00000800:00000200:12.0:1547251529.833869:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (12)-- 00000800:00000200:13.0:1547251529.833870:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (11)-- 00000800:00000200:12.0:1547251529.833870:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (11)++ 00000800:00000200:13.0:1547251529.833871:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (11)-- 00000800:00000200:12.0:1547251529.833872:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (10)-- 00000800:00000200:13.0:1547251529.833873:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (10)++ 00000800:00000200:12.0:1547251529.833874:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (10)-- 00000800:00000200:13.0:1547251529.833875:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (9)-- 00000800:00000200:12.0:1547251529.833875:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (9)++ 00000800:00000200:13.0:1547251529.833876:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (9)-- 00000800:00000200:13.0:1547251529.833877:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (8)++ 00000800:00000200:12.0:1547251529.833877:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (8)-- 00000800:00000200:12.0:1547251529.833878:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (8)-- 00000800:00000200:13.0:1547251529.833879:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (7)-- 00000800:00000200:12.0:1547251529.833879:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (7)++ 00000800:00000200:13.0:1547251529.833880:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (7)-- 00000800:00000200:13.0:1547251529.833881:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (6)++ 00000800:00000200:12.0:1547251529.833881:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (6)-- 00000800:00000200:12.0:1547251529.833882:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (6)-- 00000800:00000200:13.0:1547251529.833883:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (5)-- 00000800:00000200:12.0:1547251529.833883:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (5)++ 00000800:00000200:13.0:1547251529.833884:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (5)-- 00000800:00000200:13.0:1547251529.833885:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (4)++ 00000800:00000200:12.0:1547251529.833885:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (4)-- 00000800:00000200:12.0:1547251529.833886:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (4)-- 00000800:00000200:13.0:1547251529.833887:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (3)-- 00000800:00000200:12.0:1547251529.833887:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (3)-- 00000800:00000200:13.0:1547251529.833888:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (1)-- 00000800:00000200:9.0:1547251529.834775:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97ae6c6da580] -> 10.0.10.184@o2ib10 (1)-- 00000800:00000200:8.0:1547251529.837086:0:15956:0:(o2iblnd_cb.c:2044:kiblnd_close_conn_locked()) closing conn to 10.0.10.184@o2ib10 00000800:00000200:8.0:1547251529.837088:0:15956:0:(o2iblnd.c:417:kiblnd_unlink_peer_locked()) peer_ni[ffff97b824504c00] -> 10.0.10.184@o2ib10 (2)-- 00000800:00000200:8.0:1547251529.837094:0:15956:0:(o2iblnd_cb.c:3286:kiblnd_cm_callback()) conn[ffff97ae6a9a2800] (20)-- 00000800:00000200:0.2:1547251529.837823:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ae6a9a2800] (19)++ 00000800:00000200:12.0:1547251529.837879:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (20)++ 00000800:00000200:12.0:1547251529.837888:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (21)-- 00000800:00000200:12.0:1547251529.837889:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (20)-- 00000800:00000200:12.0:1547251529.837891:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (19)++ 00000800:00000200:12.0:1547251529.837892:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (20)-- 00000800:00000200:12.0:1547251529.837893:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (19)-- 00000800:00000200:12.0:1547251529.837894:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (18)++ 00000800:00000200:12.0:1547251529.837896:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (19)-- 00000800:00000200:12.0:1547251529.837897:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (18)-- 00000800:00000200:12.0:1547251529.837898:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (17)++ 00000800:00000200:12.0:1547251529.837899:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (18)-- 00000800:00000200:12.0:1547251529.837900:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (17)-- 00000800:00000200:12.0:1547251529.837901:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (16)++ 00000800:00000200:12.0:1547251529.837903:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (17)-- 00000800:00000200:12.0:1547251529.837904:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (16)-- 00000800:00000200:13.0:1547251529.837905:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (15)++ 00000800:00000200:13.0:1547251529.837908:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (16)-- 00000800:00000200:12.0:1547251529.837909:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (16)++ 00000800:00000200:13.0:1547251529.837910:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (16)-- 00000800:00000200:13.0:1547251529.837911:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (15)++ 00000800:00000200:12.0:1547251529.837911:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (15)-- 00000800:00000200:13.0:1547251529.837912:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (16)-- 00000800:00000200:13.0:1547251529.837913:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (15)-- 00000800:00000200:12.0:1547251529.837914:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (13)-- 00000800:00000200:13.0:1547251529.837915:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (12)++ 00000800:00000200:13.0:1547251529.837920:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (13)-- 00000800:00000200:13.0:1547251529.837921:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (12)-- 00000800:00000200:13.0:1547251529.837922:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (11)++ 00000800:00000200:13.0:1547251529.837925:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (12)-- 00000800:00000200:13.0:1547251529.837926:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (11)-- 00000800:00000200:13.0:1547251529.837927:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (10)++ 00000800:00000200:13.0:1547251529.837929:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (11)-- 00000800:00000200:12.0:1547251529.837929:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (11)++ 00000800:00000200:13.0:1547251529.837930:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (11)-- 00000800:00000200:13.0:1547251529.837931:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (10)++ 00000800:00000200:12.0:1547251529.837931:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (10)-- 00000800:00000200:12.0:1547251529.837932:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (10)-- 00000800:00000200:13.0:1547251529.837933:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (9)-- 00000800:00000200:12.0:1547251529.837933:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (9)++ 00000800:00000200:13.0:1547251529.837934:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (9)-- 00000800:00000200:13.0:1547251529.837944:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (8)++ 00000800:00000200:12.0:1547251529.837944:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (8)-- 00000800:00000200:12.0:1547251529.837945:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (8)-- 00000800:00000200:13.0:1547251529.837946:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (7)-- 00000800:00000200:12.0:1547251529.837946:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (7)++ 00000800:00000200:13.0:1547251529.837947:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (7)-- 00000800:00000200:12.0:1547251529.837949:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (6)-- 00000800:00000200:13.0:1547251529.837950:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (6)++ 00000800:00000200:12.0:1547251529.837951:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (6)-- 00000800:00000200:13.0:1547251529.837952:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (5)-- 00000800:00000200:12.0:1547251529.837952:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (5)++ 00000800:00000200:13.0:1547251529.837953:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (5)-- 00000800:00000200:12.0:1547251529.837954:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ae6a9a2800] (4)-- 00000800:00000200:13.0:1547251529.837955:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (4)-- 00000800:00000200:12.0:1547251529.837956:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ae6a9a2800] (2)-- 00000800:00000200:10.0:1547251529.838031:0:12658:0:(o2iblnd_cb.c:3533:kiblnd_connd()) conn[ffff97ae6a9a2800] (1)-- 00000800:00000200:10.0:1547251529.838824:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97b824504c00] -> 10.0.10.184@o2ib10 (1)-- 00000100:00000200:13.0:1547251529.946691:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441600, portal 4 00000100:00000200:13.0:1547251529.946694:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441600, offset 0 00000100:00000200:3.0:1547251529.946695:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441632, portal 4 00000400:00000200:13.0:1547251529.946697:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000100:00000200:3.0:1547251529.946697:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441632, offset 0 00000400:00000200:13.0:1547251529.946700:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000100:00000200:10.0:1547251529.946701:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441616, portal 4 00000400:00000200:3.0:1547251529.946701:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:13.0:1547251529.946702:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22961] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251529.946703:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000100:00000200:10.0:1547251529.946704:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441616, offset 0 00000400:00000200:13.0:1547251529.946705:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22959] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22961] 00000400:00000200:3.0:1547251529.946706:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22961] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:13.0:1547251529.946707:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:10.0:1547251529.946707:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:13.0:1547251529.946709:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251529.946709:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22960] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22961] 00000400:00000200:13.0:1547251529.946710:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251529.946710:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:13.0:1547251529.946712:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9713, 9714] 00000400:00000200:10.0:1547251529.946712:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22961] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251529.946712:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:13.0:1547251529.946714:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:3.0:1547251529.946715:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251529.946716:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22961] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22961] 00000400:00000200:3.0:1547251529.946716:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251529.946718:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251529.946718:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9714, 9714] 00000400:00000200:13.0:1547251529.946719:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:10.0:1547251529.946720:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251529.946720:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:10.0:1547251529.946721:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:13.0:1547251529.946722:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:10.0:1547251529.946723:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9714, 9715] 00000400:00000200:10.0:1547251529.946725:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:3.0:1547251529.946725:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251529.946728:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:13.0:1547251529.946729:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251529.946731:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000400:00000200:10.0:1547251529.946731:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251529.946731:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251529.946732:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:3.0:1547251529.946733:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:13.0:1547251529.946734:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:10.0:1547251529.946734:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:3.0:1547251529.946734:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:3.0:1547251529.946736:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:10.0:1547251529.946737:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:10.0:1547251529.946738:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:10.0:1547251529.946739:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:10.0:1547251529.946742:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:9.2:1547251529.946750:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:9.2:1547251529.946757:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:13.0:1547251529.946767:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:13.0:1547251529.946769:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251529.946771:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000100:00000200:13.0:1547251529.946773:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af7200 x1622384012441600/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251536 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251529.946773:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:12.0:1547251529.946777:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c6000 x1622384012441632/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251536 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251529.946780:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792bd48 00000400:00000200:13.0:1547251529.946781:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251529.946783:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555bb0 00000400:00000200:13.0:1547251529.946784:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:12.0:1547251529.946785:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:13.0:1547251529.946787:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:12.0:1547251529.946788:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251529.946789:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:13.0:1547251529.946790:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251529.946792:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:7.2:1547251529.946794:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000800:00000200:13.0:1547251529.946805:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:13.0:1547251529.946814:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000100:00000200:13.0:1547251529.946818:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960e9b00 x1622384012441616/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251536 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251529.946821:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000400:00000200:13.0:1547251529.946824:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914ee0 00000400:00000200:13.0:1547251529.946825:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251529.946828:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251529.946831:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:9.2:1547251529.946917:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:7.2:1547251529.946922:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251529.946928:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:12.0:1547251529.946930:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251529.946934:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000800:00000200:13.0:1547251529.946935:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:13.0:1547251529.946946:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251529.946947:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251529.946951:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251529.946952:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03800 00000400:00000200:13.0:1547251529.946954:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03820 00000400:00000200:12.0:1547251529.946956:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f4fd [1] + 192 00000400:00000200:13.0:1547251529.946961:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f51d [1] + 192 00000100:00000200:12.0:1547251529.946965:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af7200 x1622384012441600/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251536 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251529.946976:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:13.0:1547251529.946978:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c6000 x1622384012441632/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251536 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251529.946978:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251529.946981:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251529.946982:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251529.946984:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:13.0:1547251529.946988:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251529.946988:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.93@o2ib10 00000800:00000200:13.0:1547251529.946990:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000100:00000200:14.0:1547251529.946992:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af7200 x1622384012441600/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251536 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251529.946992:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000400:00000200:12.0:1547251529.946992:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:13.0:1547251529.946993:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:13.0:1547251529.946995:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000400:00000200:12.0:1547251529.946995:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03810 00000800:00000200:13.0:1547251529.946996:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:14.0:1547251529.946998:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af7200 x1622384012441600/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251536 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251529.946998:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:14.0:1547251529.947003:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b2a8 00000100:00000200:3.0:1547251529.947006:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c6000 x1622384012441632/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251536 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:14.0:1547251529.947007:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af7200 x1622384012441600/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251536 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251529.947011:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c6000 x1622384012441632/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251536 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251529.947016:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00000400:00000200:12.0:1547251529.947019:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f52d [1] + 192 00010000:00080000:3.0:1547251529.947019:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c6000 x1622384012441632/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251536 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:12.0:1547251529.947023:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960e9b00 x1622384012441616/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251536 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251529.947032:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251529.947034:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251529.947035:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251529.947036:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:10.0:1547251529.947078:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960e9b00 x1622384012441616/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251536 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:10.0:1547251529.947084:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960e9b00 x1622384012441616/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251536 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251529.947088:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914550 00010000:00080000:10.0:1547251529.947092:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba960e9b00 x1622384012441616/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251536 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:12.0:1547251530.835629:0:70:0:(o2iblnd_cb.c:2044:kiblnd_close_conn_locked()) closing conn to 10.2.10.184@o2ib10 00000800:00000200:12.0:1547251530.835632:0:70:0:(o2iblnd.c:417:kiblnd_unlink_peer_locked()) peer_ni[ffff97a59c828a00] -> 10.2.10.184@o2ib10 (2)-- 00000800:00000200:12.0:1547251530.835639:0:70:0:(o2iblnd_cb.c:3286:kiblnd_cm_callback()) conn[ffff97a65d02b800] (20)-- 00000800:00000200:8.2:1547251530.836067:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d02b800] (19)++ 00000800:00000200:13.0:1547251530.836118:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (20)++ 00000800:00000200:13.0:1547251530.836126:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (21)-- 00000800:00000200:13.0:1547251530.836128:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (20)-- 00000800:00000200:13.0:1547251530.836130:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (19)-- 00000800:00000200:8.2:1547251530.836217:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d02b800] (18)++ 00000800:00000200:13.0:1547251530.836231:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (19)++ 00000800:00000200:9.0:1547251530.836234:0:12658:0:(o2iblnd_cb.c:3533:kiblnd_connd()) conn[ffff97a65d02b800] (20)-- 00000800:00000200:13.0:1547251530.836237:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (19)-- 00000800:00000200:13.0:1547251530.836238:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (18)-- 00000800:00000200:13.0:1547251530.836240:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (17)++ 00000800:00000200:13.0:1547251530.836241:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (18)-- 00000800:00000200:13.0:1547251530.836242:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (17)-- 00000800:00000200:13.0:1547251530.836243:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (16)++ 00000800:00000200:13.0:1547251530.836245:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (17)-- 00000800:00000200:13.0:1547251530.836246:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (16)-- 00000800:00000200:12.0:1547251530.836251:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (15)++ 00000800:00000200:12.0:1547251530.836257:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (16)-- 00000800:00000200:12.0:1547251530.836258:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (15)-- 00000800:00000200:12.0:1547251530.836259:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (14)++ 00000800:00000200:12.0:1547251530.836263:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (15)-- 00000800:00000200:12.0:1547251530.836264:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (14)-- 00000800:00000200:12.0:1547251530.836265:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (13)++ 00000800:00000200:12.0:1547251530.836268:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (14)-- 00000800:00000200:12.0:1547251530.836270:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (13)-- 00000800:00000200:12.0:1547251530.836271:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (12)++ 00000800:00000200:12.0:1547251530.836274:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (13)-- 00000800:00000200:12.0:1547251530.836275:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (12)-- 00000800:00000200:12.0:1547251530.836276:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (11)++ 00000800:00000200:12.0:1547251530.836280:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (12)-- 00000800:00000200:12.0:1547251530.836281:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (11)-- 00000800:00000200:12.0:1547251530.836282:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (10)++ 00000800:00000200:12.0:1547251530.836285:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (11)-- 00000800:00000200:12.0:1547251530.836286:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (10)-- 00000800:00000200:12.0:1547251530.836287:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (9)++ 00000800:00000200:13.0:1547251530.836288:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (10)++ 00000800:00000200:12.0:1547251530.836288:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (10)-- 00000800:00000200:13.0:1547251530.836290:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (9)-- 00000800:00000200:12.0:1547251530.836290:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (10)-- 00000800:00000200:12.0:1547251530.836291:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (9)++ 00000800:00000200:13.0:1547251530.836292:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (9)-- 00000800:00000200:13.0:1547251530.836293:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (8)++ 00000800:00000200:12.0:1547251530.836293:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (8)-- 00000800:00000200:12.0:1547251530.836294:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (8)-- 00000800:00000200:13.0:1547251530.836295:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (7)-- 00000800:00000200:12.0:1547251530.836296:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (7)++ 00000800:00000200:13.0:1547251530.836297:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (7)-- 00000800:00000200:13.0:1547251530.836298:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (6)++ 00000800:00000200:12.0:1547251530.836298:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (6)-- 00000800:00000200:12.0:1547251530.836299:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (6)-- 00000800:00000200:13.0:1547251530.836300:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (5)-- 00000800:00000200:12.0:1547251530.836300:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (5)++ 00000800:00000200:13.0:1547251530.836301:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (5)-- 00000800:00000200:13.0:1547251530.836302:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b800] (4)++ 00000800:00000200:12.0:1547251530.836302:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (4)-- 00000800:00000200:12.0:1547251530.836303:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (4)-- 00000800:00000200:13.0:1547251530.836304:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d02b800] (3)-- 00000800:00000200:12.0:1547251530.836304:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (3)-- 00000800:00000200:13.0:1547251530.836305:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b800] (1)-- 00000800:00000200:9.0:1547251530.837196:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a00] -> 10.2.10.184@o2ib10 (1)-- 00000800:00000200:8.0:1547251530.839448:0:15956:0:(o2iblnd_cb.c:2044:kiblnd_close_conn_locked()) closing conn to 10.2.10.184@o2ib10 00000800:00000200:8.0:1547251530.839450:0:15956:0:(o2iblnd.c:417:kiblnd_unlink_peer_locked()) peer_ni[ffff97b971916900] -> 10.2.10.184@o2ib10 (2)-- 00000800:00000200:8.0:1547251530.839457:0:15956:0:(o2iblnd_cb.c:3286:kiblnd_cm_callback()) conn[ffff97b81d647800] (20)-- 00000800:00000200:0.2:1547251530.839953:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b81d647800] (19)++ 00000800:00000200:12.0:1547251530.840012:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (20)++ 00000800:00000200:12.0:1547251530.840021:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (21)-- 00000800:00000200:12.0:1547251530.840023:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (20)-- 00000800:00000200:12.0:1547251530.840024:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (19)-- 00000800:00000200:0.2:1547251530.840054:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b81d647800] (18)++ 00000800:00000200:12.0:1547251530.840066:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (19)++ 00000800:00000200:12.0:1547251530.840071:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (20)-- 00000800:00000200:12.0:1547251530.840072:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (19)-- 00000800:00000200:12.0:1547251530.840074:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (18)++ 00000800:00000200:12.0:1547251530.840076:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (19)-- 00000800:00000200:12.0:1547251530.840077:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (18)-- 00000800:00000200:12.0:1547251530.840078:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (17)++ 00000800:00000200:12.0:1547251530.840080:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (18)-- 00000800:00000200:12.0:1547251530.840081:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (17)-- 00000800:00000200:13.0:1547251530.840082:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (16)++ 00000800:00000200:13.0:1547251530.840088:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (17)-- 00000800:00000200:13.0:1547251530.840089:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (16)-- 00000800:00000200:13.0:1547251530.840090:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (15)++ 00000800:00000200:13.0:1547251530.840093:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (16)-- 00000800:00000200:13.0:1547251530.840094:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (15)-- 00000800:00000200:13.0:1547251530.840096:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (14)++ 00000800:00000200:13.0:1547251530.840099:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (15)-- 00000800:00000200:13.0:1547251530.840100:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (14)-- 00000800:00000200:13.0:1547251530.840101:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (13)++ 00000800:00000200:13.0:1547251530.840105:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (14)-- 00000800:00000200:10.0:1547251530.840105:0:12658:0:(o2iblnd_cb.c:3533:kiblnd_connd()) conn[ffff97b81d647800] (14)-- 00000800:00000200:13.0:1547251530.840106:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (13)-- 00000800:00000200:13.0:1547251530.840107:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (12)++ 00000800:00000200:13.0:1547251530.840110:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (12)-- 00000800:00000200:13.0:1547251530.840111:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (11)-- 00000800:00000200:13.0:1547251530.840113:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (10)++ 00000800:00000200:13.0:1547251530.840116:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (11)-- 00000800:00000200:13.0:1547251530.840117:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (10)-- 00000800:00000200:13.0:1547251530.840118:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (9)++ 00000800:00000200:13.0:1547251530.840121:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (10)-- 00000800:00000200:13.0:1547251530.840122:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (9)-- 00000800:00000200:13.0:1547251530.840123:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (8)++ 00000800:00000200:13.0:1547251530.840124:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (9)-- 00000800:00000200:12.0:1547251530.840125:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (9)++ 00000800:00000200:13.0:1547251530.840126:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (9)-- 00000800:00000200:13.0:1547251530.840127:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (8)++ 00000800:00000200:12.0:1547251530.840127:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (8)-- 00000800:00000200:12.0:1547251530.840129:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (8)-- 00000800:00000200:13.0:1547251530.840130:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (7)-- 00000800:00000200:12.0:1547251530.840130:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (7)++ 00000800:00000200:13.0:1547251530.840131:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (7)-- 00000800:00000200:13.0:1547251530.840132:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (6)++ 00000800:00000200:12.0:1547251530.840132:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (6)-- 00000800:00000200:12.0:1547251530.840133:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (6)-- 00000800:00000200:13.0:1547251530.840134:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (5)-- 00000800:00000200:12.0:1547251530.840134:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (5)++ 00000800:00000200:13.0:1547251530.840135:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (5)-- 00000800:00000200:13.0:1547251530.840136:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647800] (4)++ 00000800:00000200:12.0:1547251530.840136:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (4)-- 00000800:00000200:12.0:1547251530.840137:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (4)-- 00000800:00000200:13.0:1547251530.840138:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b81d647800] (3)-- 00000800:00000200:12.0:1547251530.840138:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (3)-- 00000800:00000200:13.0:1547251530.840139:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647800] (1)-- 00000800:00000200:10.0:1547251530.841099:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97b971916900] -> 10.2.10.184@o2ib10 (1)-- 00000100:00000200:4.0:1547251531.258685:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012441648, portal 25 00000100:00000200:4.0:1547251531.258689:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 26, xid 1622384012441648, offset 0 00000400:00000200:4.0:1547251531.258693:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:5.0:1547251531.258695:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012441696, portal 10 00000400:00000200:4.0:1547251531.258697:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:5.0:1547251531.258698:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012441696, offset 0 00000400:00000200:5.0:1547251531.258700:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251531.258700:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 26 MB=0x5c38cb6c03830 00000400:00000200:5.0:1547251531.258704:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251531.258704:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-0@lo of length 224/224 into md 0xd4849 [1] + 0 00000400:00000200:5.0:1547251531.258707:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c03860 00000400:00000200:4.0:1547251531.258707:0:12676:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52ddd0 00000100:00000200:4.0:1547251531.258709:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:5.0:1547251531.258710:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 55776 00000100:00000200:5.0:1547251531.258713:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:5.0:1547251531.258717:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251531.258719:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52ddd0 00000100:00000200:5.0:1547251531.258720:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c432100 x1622384012441696/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251531.258720:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251531.258722:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012441664, portal 10 00000100:00000200:6.0:1547251531.258722:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012441680, portal 10 00000400:00000200:5.0:1547251531.258726:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad6e8 00000400:00000200:5.0:1547251531.258727:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251531.258728:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251531.258728:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c432400 x1622384012441648/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251538 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251531.258734:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b908 00000400:00000200:4.0:1547251531.258735:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:7.0:1547251531.258736:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012441664, offset 0 00000400:00000200:4.0:1547251531.258736:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251531.258737:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012441680, offset 0 00000400:00000200:7.0:1547251531.258740:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251531.258742:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:4.0:1547251531.258742:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012441712, portal 10 00000400:00000200:7.0:1547251531.258744:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251531.258745:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:4.0:1547251531.258745:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012441712, offset 0 00000100:00000200:5.0:1547251531.258746:0:13338:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441696 00000400:00000200:7.0:1547251531.258748:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c03840 00000400:00000200:6.0:1547251531.258748:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c03850 00000400:00000200:4.0:1547251531.258748:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251531.258750:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251531.258751:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 56000 00010000:00000200:5.0:1547251531.258751:0:13338:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae5c435400 x1622384012441696/t0(0) o400->cache1-MDT0000-lwp-OST0002_UUID@0@lo:592/0 lens 224/224 e 0 to 0 dl 1547251537 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:6.0:1547251531.258754:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 56224 00000400:00000200:4.0:1547251531.258754:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c03870 00000100:00000200:7.0:1547251531.258757:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:7.0:1547251531.258763:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251531.258764:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 56448 00000100:00000200:6.0:1547251531.258766:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:6.0:1547251531.258770:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251531.258770:0:13338:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012441696, offset 192 00000100:00000200:7.0:1547251531.258772:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c436900 x1622384012441664/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251531.258776:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441648 00000400:00000200:7.0:1547251531.258778:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:7.0:1547251531.258779:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:4.0:1547251531.258780:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:7.0:1547251531.258781:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00010000:00000200:2.0:1547251531.258783:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2d8450 x1622384012441648/t0(0) o400->a65be517-33e7-fa36-0de3-da9e02df4f8d@0@lo:592/0 lens 224/224 e 0 to 0 dl 1547251537 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:4.0:1547251531.258785:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251531.258786:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c435d00 x1622384012441680/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251531.258789:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384012441648, offset 192 00000400:00000200:6.0:1547251531.258792:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990bb0 00000400:00000200:2.0:1547251531.258792:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251531.258793:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251531.258794:0:13338:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251531.258795:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:2.0:1547251531.258795:0:12929:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251531.258797:0:13338:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:4.0:1547251531.258797:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c436300 x1622384012441712/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251531.258797:0:12929:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 25 MB=0x5c38cb6c03830 00000400:00000200:5.0:1547251531.258801:0:13338:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c03860 00000400:00000200:2.0:1547251531.258801:0:12929:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 19 from 12345-0@lo of length 192/192 into md 0xae6e1 [1] + 192 00000400:00000200:4.0:1547251531.258802:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b770 00000400:00000200:5.0:1547251531.258803:0:13338:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xae6f1 [1] + 192 00000400:00000200:4.0:1547251531.258804:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251531.258805:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:2.0:1547251531.258808:0:12929:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c432400 x1622384012441648/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251531.258809:0:12981:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441664 00000100:00000200:6.0:1547251531.258810:0:12980:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441680 00010000:00000200:6.0:1547251531.258814:0:12980:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a765df0000 x1622384012441680/t0(0) o400->cache1-MDT0000-lwp-OST0000_UUID@0@lo:592/0 lens 224/224 e 0 to 0 dl 1547251537 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:4.0:1547251531.258814:0:13324:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441712 00010000:00000200:7.0:1547251531.258815:0:12981:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae37937500 x1622384012441664/t0(0) o400->cache1-MDT0000-lwp-MDT0000_UUID@0@lo:592/0 lens 224/224 e 0 to 0 dl 1547251537 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:2.0:1547251531.258815:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251531.258816:0:13338:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c432100 x1622384012441696/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00010000:00000200:4.0:1547251531.258819:0:13324:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444a100 x1622384012441712/t0(0) o400->cache1-MDT0000-lwp-OST0001_UUID@0@lo:592/0 lens 224/224 e 0 to 0 dl 1547251537 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:7.0:1547251531.258823:0:12981:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012441664, offset 192 00000100:00000200:6.0:1547251531.258823:0:12980:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012441680, offset 192 00000400:00000200:5.0:1547251531.258823:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:2.0:1547251531.258823:0:12929:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a198 00000400:00000200:2.0:1547251531.258824:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251531.258825:0:13338:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad6e8 00000400:00000200:2.0:1547251531.258825:0:12929:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:5.0:1547251531.258826:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251531.258827:0:12980:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251531.258827:0:13338:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251531.258828:0:12981:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251531.258830:0:12980:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:4.0:1547251531.258830:0:13324:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012441712, offset 192 00000100:00000200:2.0:1547251531.258830:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000400:00000200:7.0:1547251531.258831:0:12981:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251531.258832:0:13324:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251531.258833:0:12980:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c03850 00000400:00000200:7.0:1547251531.258835:0:12981:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c03840 00000400:00000200:4.0:1547251531.258835:0:13324:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251531.258836:0:12980:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xae711 [1] + 192 00000400:00000200:4.0:1547251531.258838:0:13324:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c03870 00000100:00000200:5.0:1547251531.258839:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c432100 x1622384012441696/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251531.258844:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c432100 x1622384012441696/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251531.258849:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adb28 00000400:00000200:7.0:1547251531.258851:0:12981:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xae701 [1] + 192 00000100:00000200:6.0:1547251531.258856:0:12980:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c435d00 x1622384012441680/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251531.258863:0:12980:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251531.258864:0:12980:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990bb0 00000400:00000200:4.0:1547251531.258864:0:13324:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xae721 [1] + 192 00000400:00000200:6.0:1547251531.258865:0:12980:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:7.0:1547251531.258867:0:12981:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c436900 x1622384012441664/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251531.258867:0:12980:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251531.258874:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251531.258875:0:13324:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c436300 x1622384012441712/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251531.258876:0:12981:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:7.0:1547251531.258876:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251531.258878:0:12981:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251531.258879:0:13324:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251531.258880:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c435d00 x1622384012441680/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251531.258881:0:13324:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b770 00000400:00000200:4.0:1547251531.258881:0:13324:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251531.258882:0:13324:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251531.258886:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c435d00 x1622384012441680/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251531.258891:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00000100:00000200:7.0:1547251531.258893:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c436900 x1622384012441664/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251531.258898:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c436900 x1622384012441664/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251531.258904:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00000100:00000200:4.0:1547251531.258906:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c436300 x1622384012441712/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251531.258910:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c436300 x1622384012441712/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251531.258913:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b908 00000100:00000200:4.0:1547251531.258916:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c432400 x1622384012441648/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251531.258920:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c432400 x1622384012441648/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251538 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251531.258924:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040ba18 00000100:00000200:15.0:1547251532.058730:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441728, portal 4 00000100:00000200:15.0:1547251532.058734:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441728, offset 0 00000100:00000200:0.0:1547251532.058735:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441744, portal 4 00000400:00000200:15.0:1547251532.058738:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000100:00000200:0.0:1547251532.058738:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441744, offset 0 00000400:00000200:15.0:1547251532.058741:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:0.0:1547251532.058741:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:15.0:1547251532.058743:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22962] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251532.058744:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:15.0:1547251532.058747:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22961] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22962] 00000400:00000200:0.0:1547251532.058747:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22962] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251532.058749:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251532.058750:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22961] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22962] 00000400:00000200:15.0:1547251532.058751:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251532.058752:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251532.058752:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:15.0:1547251532.058754:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9719, 9720] 00000400:00000200:15.0:1547251532.058756:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:0.0:1547251532.058757:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251532.058758:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251532.058760:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000400:00000200:0.0:1547251532.058760:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9720, 9720] 00000400:00000200:0.0:1547251532.058761:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000800:00000200:15.0:1547251532.058763:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:0.0:1547251532.058766:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251532.058767:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251532.058769:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:0.0:1547251532.058769:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:15.0:1547251532.058770:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:0.0:1547251532.058772:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251532.058773:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000800:00000200:0.0:1547251532.058773:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000100:00000200:9.0:1547251532.058774:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441760, portal 4 00000800:00000200:0.0:1547251532.058774:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000100:00000200:9.0:1547251532.058777:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441760, offset 0 00000800:00000200:0.0:1547251532.058777:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000400:00000200:9.0:1547251532.058780:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251532.058783:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251532.058786:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22963] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:7.2:1547251532.058788:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000400:00000200:9.0:1547251532.058789:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22962] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22963] 00000400:00000200:9.0:1547251532.058791:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251532.058793:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251532.058794:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251532.058795:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9720, 9721] 00000400:00000200:9.0:1547251532.058797:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000800:00000200:10.0:1547251532.058801:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000400:00000200:9.0:1547251532.058802:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251532.058804:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:9.0:1547251532.058807:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251532.058809:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (23)++ 00000800:00000200:10.0:1547251532.058810:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000800:00000200:9.0:1547251532.058810:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (24)++ 00000800:00000200:9.0:1547251532.058813:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (24)-- 00000100:00000200:10.0:1547251532.058814:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af4200 x1622384012441728/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251532.058822:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1660 00000400:00000200:10.0:1547251532.058824:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251532.058826:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:11.0:1547251532.058828:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000400:00000200:10.0:1547251532.058828:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:5.2:1547251532.058828:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251532.058831:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000100:00000200:11.0:1547251532.058833:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf6600 x1622384012441760/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251532.058833:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251532.058835:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251532.058837:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000400:00000200:11.0:1547251532.058839:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea7f8 00000400:00000200:11.0:1547251532.058840:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:10.0:1547251532.058840:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c5d00 x1622384012441744/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251532.058844:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251532.058846:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000400:00000200:10.0:1547251532.058846:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665880 00000400:00000200:10.0:1547251532.058847:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251532.058848:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251532.058850:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251532.058852:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:7.2:1547251532.058908:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:11.0:1547251532.058920:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251532.058928:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251532.058931:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251532.058932:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251532.058933:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251532.058934:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:7.2:1547251532.058948:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:5.2:1547251532.058952:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251532.058958:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:10.0:1547251532.058960:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251532.058961:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251532.058963:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251532.058964:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251532.058968:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251532.058968:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251532.058973:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c038a0 00000400:00000200:10.0:1547251532.058973:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03880 00000400:00000200:11.0:1547251532.058977:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6329 [1] + 192 00000400:00000200:10.0:1547251532.058983:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6309 [1] + 192 00000100:00000200:11.0:1547251532.058986:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf6600 x1622384012441760/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251532.058997:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251532.058998:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af4200 x1622384012441728/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251532.058999:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251532.059002:0:12663:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (23)++ 00000800:00000200:11.0:1547251532.059004:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (24)-- 00000800:00000200:11.0:1547251532.059005:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251532.059007:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:10.0:1547251532.059008:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251532.059010:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251532.059010:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000100:00000200:9.0:1547251532.059010:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf6600 x1622384012441760/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251532.059012:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251532.059013:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:11.0:1547251532.059014:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251532.059016:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:9.0:1547251532.059016:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf6600 x1622384012441760/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251532.059017:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03890 00000800:00000200:10.0:1547251532.059017:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000800:00000200:10.0:1547251532.059019:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251532.059020:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:9.0:1547251532.059020:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eae58 00000800:00000200:10.0:1547251532.059021:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000100:00000200:15.0:1547251532.059023:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af4200 x1622384012441728/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251532.059023:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00010000:00080000:9.0:1547251532.059024:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf6600 x1622384012441760/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:15.0:1547251532.059029:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af4200 x1622384012441728/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251532.059033:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00000400:00000200:11.0:1547251532.059036:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6319 [1] + 192 00010000:00080000:15.0:1547251532.059037:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af4200 x1622384012441728/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251532.059040:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c5d00 x1622384012441744/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251532.059049:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251532.059051:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251532.059053:0:12663:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:11.0:1547251532.059054:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:11.0:1547251532.059055:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000100:00000200:0.0:1547251532.059059:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c5d00 x1622384012441744/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251532.059064:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c5d00 x1622384012441744/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:7.2:1547251532.059065:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:0.0:1547251532.059068:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00010000:00080000:0.0:1547251532.059072:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c5d00 x1622384012441744/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:10.0:1547251532.059079:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251532.059083:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251532.059085:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251532.059086:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:8.0:1547251532.522684:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441776, portal 4 00000100:00000200:8.0:1547251532.522686:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441776, offset 0 00000400:00000200:8.0:1547251532.522689:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251532.522692:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251532.522694:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22963] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:14.0:1547251532.522697:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441792, portal 4 00000400:00000200:8.0:1547251532.522698:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22963] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22963] 00000100:00000200:14.0:1547251532.522700:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441792, offset 0 00000400:00000200:8.0:1547251532.522701:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251532.522702:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251532.522703:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251532.522704:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251532.522705:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9713, 9713] 00000400:00000200:14.0:1547251532.522706:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251532.522707:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:14.0:1547251532.522709:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22964] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251532.522711:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:14.0:1547251532.522712:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22963] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22964] 00000400:00000200:14.0:1547251532.522714:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:8.0:1547251532.522714:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:14.0:1547251532.522716:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251532.522717:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:14.0:1547251532.522719:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251532.522719:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000400:00000200:14.0:1547251532.522720:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9713, 9714] 00000800:00000200:8.0:1547251532.522720:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000400:00000200:14.0:1547251532.522722:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:8.0:1547251532.522723:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000400:00000200:14.0:1547251532.522727:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251532.522730:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:14.0:1547251532.522733:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251532.522734:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:14.0:1547251532.522735:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000100:00000200:3.0:1547251532.522736:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441808, portal 4 00000800:00000200:14.0:1547251532.522738:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:3.0:1547251532.522739:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441808, offset 0 00000800:00000200:12.2:1547251532.522741:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000400:00000200:3.0:1547251532.522742:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:3.0:1547251532.522745:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:3.0:1547251532.522747:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22964] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251532.522750:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22964] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22964] 00000400:00000200:3.0:1547251532.522753:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251532.522754:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251532.522756:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251532.522757:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9714, 9714] 00000800:00000200:0.2:1547251532.522758:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:3.0:1547251532.522759:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:3.0:1547251532.522763:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251532.522766:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:3.0:1547251532.522769:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251532.522772:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (22)++ 00000800:00000200:3.0:1547251532.522773:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00000800:00000200:3.0:1547251532.522776:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (24)-- 00000800:00000200:7.0:1547251532.522796:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:7.0:1547251532.522799:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000100:00000200:7.0:1547251532.522803:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824751200 x1622384012441776/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251532.522808:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000400:00000200:7.0:1547251532.522810:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251532.522812:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:7.0:1547251532.522814:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:7.0:1547251532.522816:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:7.0:1547251532.522818:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251532.522818:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000100:00000200:7.0:1547251532.522820:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a2d00 x1622384012441792/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251532.522821:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000400:00000200:7.0:1547251532.522824:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf606e8 00000100:00000200:4.0:1547251532.522824:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c5a00 x1622384012441808/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251532.522825:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251532.522827:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:7.0:1547251532.522829:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:4.0:1547251532.522830:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555ee0 00000800:00000200:7.0:1547251532.522831:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000400:00000200:4.0:1547251532.522831:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:7.0:1547251532.522832:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:4.0:1547251532.522834:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251532.522836:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:12.2:1547251532.522866:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:7.0:1547251532.522878:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:7.0:1547251532.522887:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.0.11.92@o2ib10 00000800:00000200:7.0:1547251532.522888:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000800:00000200:7.0:1547251532.522890:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:7.0:1547251532.522890:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:7.0:1547251532.522893:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:0.2:1547251532.522896:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:12.2:1547251532.522901:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:4.0:1547251532.522907:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:7.0:1547251532.522908:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:7.0:1547251532.522910:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251532.522910:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251532.522914:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251532.522914:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:7.0:1547251532.522919:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c038c0 00000400:00000200:4.0:1547251532.522919:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c038b0 00000400:00000200:7.0:1547251532.522922:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3045 [1] + 192 00000400:00000200:4.0:1547251532.522926:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3035 [1] + 192 00000100:00000200:7.0:1547251532.522930:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a2d00 x1622384012441792/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251532.522945:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:7.0:1547251532.522947:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000100:00000200:4.0:1547251532.522947:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824751200 x1622384012441776/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:7.0:1547251532.522949:0:12662:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (23)++ 00000800:00000200:7.0:1547251532.522951:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (24)-- 00000800:00000200:7.0:1547251532.522952:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (23)-- 00000800:00000200:7.0:1547251532.522954:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:7.0:1547251532.522956:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251532.522958:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251532.522958:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:14.0:1547251532.522960:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a2d00 x1622384012441792/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251532.522960:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)++ 00000400:00000200:7.0:1547251532.522961:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c038d0 00000800:00000200:4.0:1547251532.522963:0:12661:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (24)++ 00000800:00000200:4.0:1547251532.522965:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (25)-- 00000100:00000200:14.0:1547251532.522966:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a2d00 x1622384012441792/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251532.522966:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (24)-- 00000800:00000200:4.0:1547251532.522968:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251532.522969:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000400:00000200:14.0:1547251532.522970:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60088 00000800:00000200:4.0:1547251532.522970:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251532.522971:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (23)++ 00000400:00000200:7.0:1547251532.522972:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3055 [1] + 192 00000800:00000200:4.0:1547251532.522972:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (24)-- 00000800:00000200:4.0:1547251532.522973:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (23)-- 00010000:00080000:14.0:1547251532.522974:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a2d00 x1622384012441792/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:4.0:1547251532.522974:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:4.0:1547251532.522975:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000100:00000200:8.0:1547251532.522976:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824751200 x1622384012441776/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251532.522982:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824751200 x1622384012441776/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251532.522986:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688d48 00000100:00000200:7.0:1547251532.522988:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c5a00 x1622384012441808/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251532.522989:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824751200 x1622384012441776/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:7.0:1547251532.522995:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:7.0:1547251532.522997:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:7.0:1547251532.522998:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:7.0:1547251532.522999:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000100:00000200:3.0:1547251532.523005:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c5a00 x1622384012441808/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251532.523010:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c5a00 x1622384012441808/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251532.523014:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251532.523018:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c5a00 x1622384012441808/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251532.922679:0:12678:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441824, portal 4 00000100:00000200:11.0:1547251532.922682:0:12678:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441824, offset 0 00000400:00000200:11.0:1547251532.922684:0:12678:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:11.0:1547251532.922687:0:12678:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:11.0:1547251532.922690:0:12678:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c038e0 00000400:00000200:11.0:1547251532.922692:0:12678:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd3d49 [1] + 0 00000400:00000200:11.0:1547251532.922695:0:12678:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ae6a914f68 00000100:00000200:11.0:1547251532.922696:0:12678:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:11.0:1547251532.922705:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914f68 00000400:00000200:11.0:1547251532.922706:0:12678:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251532.922708:0:12678:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07ad00 x1622384012441824/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251532.922709:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441840, portal 4 00000100:00000200:7.0:1547251532.922712:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441840, offset 0 00000400:00000200:11.0:1547251532.922713:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0baa0 00000400:00000200:11.0:1547251532.922714:0:12678:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251532.922714:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:11.0:1547251532.922715:0:12678:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251532.922717:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251532.922721:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c038f0 00000400:00000200:7.0:1547251532.922723:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe16a5 [1] + 0 00000400:00000200:7.0:1547251532.922725:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990c38 00000100:00000200:7.0:1547251532.922727:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:9.0:1547251532.922735:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441824 00000400:00000200:7.0:1547251532.922736:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990c38 00000400:00000200:7.0:1547251532.922737:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251532.922740:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df2100 x1622384012441840/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00010000:00000200:9.0:1547251532.922743:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3df450 x1622384012441824/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:593/0 lens 224/368 e 0 to 0 dl 1547251538 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:7.0:1547251532.922744:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:7.0:1547251532.922745:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251532.922746:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251532.922750:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012441824, offset 192 00000400:00000200:9.0:1547251532.922753:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251532.922756:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251532.922758:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c038e0 00000400:00000200:9.0:1547251532.922761:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae761 [1] + 192 00000100:00000200:9.0:1547251532.922764:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07ad00 x1622384012441824/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251532.922767:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441840 00000400:00000200:9.0:1547251532.922773:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251532.922775:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eae58 00000400:00000200:9.0:1547251532.922776:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251532.922777:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00010000:00000200:5.0:1547251532.922777:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11c450 x1622384012441840/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:593/0 lens 224/368 e 0 to 0 dl 1547251538 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:11.0:1547251532.922781:0:12678:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07ad00 x1622384012441824/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251532.922782:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251532.922784:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012441840, offset 192 00000100:00000200:11.0:1547251532.922786:0:12678:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07ad00 x1622384012441824/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251532.922787:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:11.0:1547251532.922790:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bb28 00000400:00000200:5.0:1547251532.922790:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00010000:00080000:11.0:1547251532.922793:0:12678:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b17b07ad00 x1622384012441824/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251532.922793:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c038f0 00000400:00000200:5.0:1547251532.922795:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae771 [1] + 192 00000100:00000200:5.0:1547251532.922799:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df2100 x1622384012441840/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251532.922808:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251532.922810:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adb28 00000400:00000200:5.0:1547251532.922811:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251532.922812:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251532.922816:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df2100 x1622384012441840/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251532.922817:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251532.922821:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df2100 x1622384012441840/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251532.922824:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00010000:00080000:7.0:1547251532.922827:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df2100 x1622384012441840/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251532.938678:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441856, portal 4 00000100:00000200:6.0:1547251532.938681:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441856, offset 0 00000400:00000200:6.0:1547251532.938684:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251532.938687:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251532.938689:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03900 00000400:00000200:6.0:1547251532.938692:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe16d5 [1] + 0 00000400:00000200:6.0:1547251532.938694:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040bdd0 00000100:00000200:6.0:1547251532.938696:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251532.938704:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bdd0 00000400:00000200:6.0:1547251532.938705:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251532.938708:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37933f00 x1622384012441856/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251532.938712:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990bb0 00000400:00000200:6.0:1547251532.938713:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251532.938714:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251532.938735:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012441856 00010000:00000200:4.0:1547251532.938744:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11c050 x1622384012441856/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:593/0 lens 224/368 e 0 to 0 dl 1547251538 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251532.938751:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012441856, offset 192 00000400:00000200:4.0:1547251532.938753:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251532.938756:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251532.938759:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03900 00000400:00000200:4.0:1547251532.938761:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae781 [1] + 192 00000100:00000200:4.0:1547251532.938765:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37933f00 x1622384012441856/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251532.938774:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251532.938776:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040ba18 00000400:00000200:4.0:1547251532.938776:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251532.938777:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251532.938781:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37933f00 x1622384012441856/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251532.938782:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251532.938785:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37933f00 x1622384012441856/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251532.938789:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00010000:00080000:6.0:1547251532.938792:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37933f00 x1622384012441856/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251539 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251534.954693:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441872, portal 4 00000100:00000200:9.0:1547251534.954696:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441872, offset 0 00000400:00000200:9.0:1547251534.954699:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:9.0:1547251534.954702:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:9.0:1547251534.954704:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22965] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:15.0:1547251534.954708:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441888, portal 4 00000400:00000200:9.0:1547251534.954708:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22964] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22965] 00000400:00000200:9.0:1547251534.954710:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000100:00000200:15.0:1547251534.954711:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441888, offset 0 00000400:00000200:9.0:1547251534.954711:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251534.954713:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:2.0:1547251534.954713:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441904, portal 4 00000400:00000200:15.0:1547251534.954714:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:9.0:1547251534.954714:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9714, 9716] 00000400:00000200:9.0:1547251534.954716:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000100:00000200:2.0:1547251534.954716:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441904, offset 0 00000400:00000200:15.0:1547251534.954717:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:2.0:1547251534.954719:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:15.0:1547251534.954720:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22965] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251534.954722:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:15.0:1547251534.954723:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22965] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22965] 00000400:00000200:2.0:1547251534.954724:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22965] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251534.954725:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:15.0:1547251534.954727:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251534.954727:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22965] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22965] 00000400:00000200:15.0:1547251534.954729:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251534.954730:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9715, 9716] 00000400:00000200:2.0:1547251534.954730:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251534.954731:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251534.954732:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:2.0:1547251534.954733:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251534.954734:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9716, 9716] 00000400:00000200:2.0:1547251534.954736:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:15.0:1547251534.954737:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251534.954739:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:9.0:1547251534.954742:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251534.954745:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:9.0:1547251534.954748:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000400:00000200:2.0:1547251534.954749:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251534.954750:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:2.0:1547251534.954751:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:15.0:1547251534.954752:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251534.954752:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:15.0:1547251534.954754:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:15.0:1547251534.954755:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:9.0:1547251534.954755:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:2.0:1547251534.954755:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251534.954756:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:2.0:1547251534.954757:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251534.954758:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:2.0:1547251534.954760:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:9.2:1547251534.954763:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:9.2:1547251534.954770:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251534.954778:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251534.954783:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000100:00000200:12.0:1547251534.954787:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07b900 x1622384012441872/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251541 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251534.954792:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:13.0:1547251534.954794:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000400:00000200:12.0:1547251534.954795:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea220 00000100:00000200:13.0:1547251534.954798:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f7800 x1622384012441904/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251541 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251534.954798:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251534.954802:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:13.0:1547251534.954804:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a3b8 00000400:00000200:13.0:1547251534.954805:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:12.0:1547251534.954805:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251534.954807:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:7.2:1547251534.954808:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:13.0:1547251534.954809:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251534.954811:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:13.0:1547251534.954812:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:12.0:1547251534.954813:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251534.954820:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:13.0:1547251534.954823:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:12.0:1547251534.954824:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8caef00 x1622384012441888/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251541 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251534.954830:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb16e8 00000400:00000200:12.0:1547251534.954831:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251534.954834:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251534.954837:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:7.2:1547251534.954924:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:9.2:1547251534.954931:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:7.2:1547251534.954942:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:13.0:1547251534.954947:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251534.954950:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:13.0:1547251534.954951:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.1.11.93@o2ib10 00000800:00000200:12.0:1547251534.954952:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000400:00000200:13.0:1547251534.954956:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251534.954957:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251534.954961:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03910 00000400:00000200:12.0:1547251534.954961:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03920 00000400:00000200:12.0:1547251534.954965:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f56d [1] + 192 00000400:00000200:13.0:1547251534.954969:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f55d [1] + 192 00000100:00000200:12.0:1547251534.954973:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8caef00 x1622384012441888/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251541 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251534.954984:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:13.0:1547251534.954986:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07b900 x1622384012441872/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251541 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251534.954986:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251534.954989:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251534.954990:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251534.954992:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251534.954994:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:13.0:1547251534.954997:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:15.0:1547251534.954999:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8caef00 x1622384012441888/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251541 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251534.954999:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000400:00000200:12.0:1547251534.954999:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:13.0:1547251534.955001:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:13.0:1547251534.955002:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:12.0:1547251534.955003:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03930 00000800:00000200:13.0:1547251534.955004:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000100:00000200:15.0:1547251534.955005:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8caef00 x1622384012441888/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251541 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251534.955005:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:13.0:1547251534.955007:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000400:00000200:15.0:1547251534.955014:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00000100:00000200:9.0:1547251534.955017:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07b900 x1622384012441872/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251541 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:15.0:1547251534.955018:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8caef00 x1622384012441888/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251541 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251534.955023:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07b900 x1622384012441872/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251541 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251534.955027:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaa18 00000400:00000200:12.0:1547251534.955030:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f58d [1] + 192 00010000:00080000:9.0:1547251534.955031:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b17b07b900 x1622384012441872/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251541 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:12.0:1547251534.955034:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f7800 x1622384012441904/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251541 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251534.955043:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251534.955045:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251534.955046:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251534.955047:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:2.0:1547251534.955088:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f7800 x1622384012441904/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251541 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251534.955094:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f7800 x1622384012441904/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251541 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251534.955098:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00010000:00080000:2.0:1547251534.955102:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f7800 x1622384012441904/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251541 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251537.066682:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441920, portal 4 00000100:00000200:8.0:1547251537.066685:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441920, offset 0 00000400:00000200:8.0:1547251537.066688:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251537.066690:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251537.066693:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22966] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251537.066696:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22966] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22966] 00000400:00000200:8.0:1547251537.066698:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251537.066700:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251537.066701:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:14.0:1547251537.066702:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441936, portal 4 00000400:00000200:8.0:1547251537.066702:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9721, 9721] 00000100:00000200:1.0F:1547251537.066702:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441952, portal 4 00000400:00000200:8.0:1547251537.066704:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000100:00000200:14.0:1547251537.066705:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441936, offset 0 00000100:00000200:1.0:1547251537.066706:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441952, offset 0 00000400:00000200:14.0:1547251537.066708:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:1.0:1547251537.066709:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251537.066710:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000400:00000200:14.0:1547251537.066711:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:8.0:1547251537.066712:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:1.0:1547251537.066712:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:14.0:1547251537.066713:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22967] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:8.0:1547251537.066715:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000400:00000200:1.0:1547251537.066715:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22967] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:14.0:1547251537.066717:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22966] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22967] 00000800:00000200:8.0:1547251537.066717:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:8.0:1547251537.066718:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000400:00000200:1.0:1547251537.066718:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22966] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22967] 00000400:00000200:14.0:1547251537.066719:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:14.0:1547251537.066721:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251537.066721:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000400:00000200:1.0:1547251537.066721:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:14.0:1547251537.066722:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251537.066722:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251537.066724:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9721, 9722] 00000400:00000200:1.0:1547251537.066724:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251537.066726:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:1.0:1547251537.066726:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9721, 9722] 00000400:00000200:1.0:1547251537.066728:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:14.0:1547251537.066731:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251537.066734:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:14.0:1547251537.066737:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000400:00000200:1.0:1547251537.066737:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251537.066739:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:14.0:1547251537.066740:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:1.0:1547251537.066740:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:14.0:1547251537.066743:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000800:00000200:1.0:1547251537.066743:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251537.066744:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (21)++ 00000800:00000200:1.0:1547251537.066745:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (22)++ 00000800:00000200:1.0:1547251537.066748:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (23)-- 00000800:00000200:5.2:1547251537.066771:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:7.2:1547251537.066792:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251537.066803:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251537.066808:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251537.066809:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:11.0:1547251537.066811:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000100:00000200:10.0:1547251537.066813:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824753f00 x1622384012441920/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251537.066815:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a4b00 x1622384012441936/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251537.066820:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000400:00000200:11.0:1547251537.066821:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:11.0:1547251537.066822:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251537.066822:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251537.066825:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251537.066826:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251537.066828:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251537.066829:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251537.066830:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:11.0:1547251537.066831:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251537.066835:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251537.066836:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000100:00000200:11.0:1547251537.066838:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b4800 x1622384012441952/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251537.066844:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2ee0 00000400:00000200:11.0:1547251537.066845:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:5.2:1547251537.066846:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000400:00000200:11.0:1547251537.066848:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251537.066851:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:11.0:1547251537.066853:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251537.066856:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251537.066858:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251537.066858:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251537.066860:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251537.066861:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:5.2:1547251537.066886:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251537.066891:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:5.2:1547251537.066896:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:10.0:1547251537.066899:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251537.066902:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251537.066903:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251537.066906:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251537.066907:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251537.066910:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251537.066912:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03940 00000400:00000200:11.0:1547251537.066914:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03950 00000400:00000200:10.0:1547251537.066916:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6399 [1] + 192 00000400:00000200:11.0:1547251537.066920:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd63b9 [1] + 192 00000100:00000200:10.0:1547251537.066924:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824753f00 x1622384012441920/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251537.066945:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251537.066946:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a4b00 x1622384012441936/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251537.066947:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251537.066949:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:10.0:1547251537.066950:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251537.066952:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251537.066954:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251537.066957:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251537.066959:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000400:00000200:10.0:1547251537.066960:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000100:00000200:8.0:1547251537.066960:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824753f00 x1622384012441920/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251537.066961:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251537.066962:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:11.0:1547251537.066964:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000400:00000200:10.0:1547251537.066964:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03960 00000100:00000200:8.0:1547251537.066965:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824753f00 x1622384012441920/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251537.066966:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:11.0:1547251537.066967:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:8.0:1547251537.066969:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688d48 00000100:00000200:14.0:1547251537.066973:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a4b00 x1622384012441936/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251537.066973:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824753f00 x1622384012441920/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:14.0:1547251537.066978:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a4b00 x1622384012441936/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251537.066983:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60088 00010000:00080000:14.0:1547251537.066986:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a4b00 x1622384012441936/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251537.066986:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd63c9 [1] + 192 00000100:00000200:10.0:1547251537.066990:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b4800 x1622384012441952/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251537.067000:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251537.067001:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251537.067002:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251537.067003:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:1.0:1547251537.067010:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b4800 x1622384012441952/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:1.0:1547251537.067016:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b4800 x1622384012441952/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251537.067020:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00010000:00080000:1.0:1547251537.067024:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b4800 x1622384012441952/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251537.530696:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441984, portal 4 00000100:00000200:9.0:1547251537.530699:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441984, offset 0 00000400:00000200:9.0:1547251537.530702:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251537.530704:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251537.530707:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22967] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251537.530710:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22968] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22967] 00000400:00000200:9.0:1547251537.530712:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251537.530714:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251537.530715:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251537.530716:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9714, 9715] 00000400:00000200:9.0:1547251537.530718:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:9.0:1547251537.530723:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251537.530726:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:9.0:1547251537.530728:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251537.530730:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:9.0:1547251537.530731:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:9.0:1547251537.530733:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000100:00000200:13.0:1547251537.530739:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012441968, portal 4 00000100:00000200:13.0:1547251537.530742:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012441968, offset 0 00000400:00000200:13.0:1547251537.530748:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:12.2:1547251537.530751:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000400:00000200:13.0:1547251537.530756:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:13.0:1547251537.530759:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22968] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:13.0:1547251537.530764:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22968] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22968] 00000400:00000200:13.0:1547251537.530767:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:13.0:1547251537.530769:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251537.530771:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251537.530773:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9715, 9715] 00000400:00000200:13.0:1547251537.530776:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:13.0:1547251537.530783:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251537.530787:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:13.0:1547251537.530791:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251537.530792:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:13.0:1547251537.530794:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000100:00000200:0.0:1547251537.530794:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442000, portal 4 00000800:00000200:13.0:1547251537.530796:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000100:00000200:0.0:1547251537.530798:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442000, offset 0 00000800:00000200:0.2:1547251537.530804:0:12670:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251537.530814:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251537.530818:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000400:00000200:0.0:1547251537.530819:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000100:00000200:4.0:1547251537.530821:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf6900 x1622384012441984/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251537.530822:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:0.0:1547251537.530825:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22968] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:4.0:1547251537.530827:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea5d8 00000400:00000200:4.0:1547251537.530828:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:0.0:1547251537.530828:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22969] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22968] 00000400:00000200:0.0:1547251537.530830:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:4.0:1547251537.530831:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:0.0:1547251537.530832:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:4.0:1547251537.530833:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:0.0:1547251537.530833:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251537.530834:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9715, 9716] 00000800:00000200:4.0:1547251537.530835:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251537.530836:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000400:00000200:0.0:1547251537.530836:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000100:00000200:4.0:1547251537.530838:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8ca9500 x1622384012441968/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251537.530841:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:4.0:1547251537.530842:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b3b8 00000400:00000200:4.0:1547251537.530843:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:0.0:1547251537.530843:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:4.0:1547251537.530845:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251537.530846:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:0.0:1547251537.530847:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:4.0:1547251537.530848:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:4.0:1547251537.530849:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:0.0:1547251537.530849:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:0.0:1547251537.530850:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:0.0:1547251537.530853:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:12.2:1547251537.530866:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251537.530880:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251537.530888:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000100:00000200:4.0:1547251537.530890:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f7b00 x1622384012442000/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251537.530894:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665aa0 00000800:00000200:7.0:1547251537.530895:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:4.0:1547251537.530895:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251537.530897:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251537.530898:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:12.2:1547251537.530916:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:7.0:1547251537.530927:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:0.2:1547251537.530935:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:7.0:1547251537.530943:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.92@o2ib10 00000400:00000200:7.0:1547251537.530947:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251537.530951:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000400:00000200:7.0:1547251537.530952:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03980 00000800:00000200:4.0:1547251537.530954:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:7.0:1547251537.530955:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe30c5 [1] + 192 00000400:00000200:4.0:1547251537.530957:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000100:00000200:7.0:1547251537.530959:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf6900 x1622384012441984/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251537.530961:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03990 00000400:00000200:7.0:1547251537.530968:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:7.0:1547251537.530970:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (23)++ 00000400:00000200:4.0:1547251537.530970:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe30e5 [1] + 192 00000800:00000200:7.0:1547251537.530972:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (24)-- 00000800:00000200:7.0:1547251537.530973:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (23)-- 00000100:00000200:4.0:1547251537.530973:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f7b00 x1622384012442000/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:7.0:1547251537.530975:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:7.0:1547251537.530977:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251537.530980:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:7.0:1547251537.530982:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03970 00000400:00000200:4.0:1547251537.530986:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251537.530988:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000100:00000200:9.0:1547251537.530989:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf6900 x1622384012441984/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251537.530990:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:4.0:1547251537.530991:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251537.530992:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:4.0:1547251537.530993:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000100:00000200:9.0:1547251537.530994:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf6900 x1622384012441984/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251537.530998:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaa18 00000400:00000200:7.0:1547251537.531002:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe30d5 [1] + 192 00010000:00080000:9.0:1547251537.531003:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf6900 x1622384012441984/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:0.0:1547251537.531005:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f7b00 x1622384012442000/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251537.531011:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f7b00 x1622384012442000/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251537.531015:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00000100:00000200:7.0:1547251537.531018:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8ca9500 x1622384012441968/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251537.531019:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f7b00 x1622384012442000/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:7.0:1547251537.531027:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:7.0:1547251537.531028:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:7.0:1547251537.531030:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:7.0:1547251537.531030:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000100:00000200:13.0:1547251537.531038:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8ca9500 x1622384012441968/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:13.0:1547251537.531043:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8ca9500 x1622384012441968/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251537.531048:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792bd48 00010000:00080000:13.0:1547251537.531051:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8ca9500 x1622384012441968/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251537.930709:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442016, portal 4 00000100:00000200:6.0:1547251537.930711:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442016, offset 0 00000400:00000200:6.0:1547251537.930714:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251537.930717:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251537.930719:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c039a0 00000400:00000200:6.0:1547251537.930722:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1755 [1] + 0 00000100:00000200:11.0:1547251537.930724:0:12678:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442032, portal 4 00000400:00000200:6.0:1547251537.930724:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040be58 00000100:00000200:11.0:1547251537.930727:0:12678:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442032, offset 0 00000100:00000200:6.0:1547251537.930727:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:11.0:1547251537.930729:0:12678:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:11.0:1547251537.930732:0:12678:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:11.0:1547251537.930735:0:12678:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c039b0 00000400:00000200:6.0:1547251537.930735:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040be58 00000400:00000200:6.0:1547251537.930736:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:11.0:1547251537.930738:0:12678:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd3db9 [1] + 0 00000100:00000200:6.0:1547251537.930738:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37930900 x1622384012442016/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251537.930741:0:12678:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ae6a914110 00000100:00000200:11.0:1547251537.930742:0:12678:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251537.930743:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990bb0 00000400:00000200:6.0:1547251537.930744:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251537.930745:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:11.0:1547251537.930750:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914110 00000400:00000200:11.0:1547251537.930751:0:12678:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251537.930753:0:12678:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07cb00 x1622384012442032/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251537.930758:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0baa0 00000400:00000200:11.0:1547251537.930759:0:12678:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251537.930760:0:12678:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251537.930766:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442016 00010000:00000200:4.0:1547251537.930775:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11bc50 x1622384012442016/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:598/0 lens 224/368 e 0 to 0 dl 1547251543 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:8.0:1547251537.930781:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442032 00000100:00000200:4.0:1547251537.930782:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442016, offset 192 00000400:00000200:4.0:1547251537.930784:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251537.930788:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00010000:00000200:8.0:1547251537.930789:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3df050 x1622384012442032/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:598/0 lens 224/368 e 0 to 0 dl 1547251543 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:4.0:1547251537.930790:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c039a0 00000400:00000200:4.0:1547251537.930793:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae7c1 [1] + 192 00000100:00000200:8.0:1547251537.930796:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442032, offset 192 00000100:00000200:4.0:1547251537.930796:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37930900 x1622384012442016/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251537.930799:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251537.930802:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251537.930804:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c039b0 00000400:00000200:4.0:1547251537.930805:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251537.930807:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae7d1 [1] + 192 00000400:00000200:4.0:1547251537.930807:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b908 00000400:00000200:4.0:1547251537.930808:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251537.930809:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251537.930810:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07cb00 x1622384012442032/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251537.930814:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:8.0:1547251537.930819:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251537.930821:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688d48 00000100:00000200:6.0:1547251537.930821:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37930900 x1622384012442016/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251537.930822:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251537.930823:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251537.930826:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37930900 x1622384012442016/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251537.930828:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:6.0:1547251537.930829:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00000100:00000200:11.0:1547251537.930831:0:12678:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07cb00 x1622384012442032/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:6.0:1547251537.930832:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37930900 x1622384012442016/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251537.930836:0:12678:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07cb00 x1622384012442032/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251537.930840:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bb28 00010000:00080000:11.0:1547251537.930843:0:12678:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b17b07cb00 x1622384012442032/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251537.946697:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442048, portal 4 00000100:00000200:7.0:1547251537.946700:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442048, offset 0 00000400:00000200:7.0:1547251537.946703:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251537.946706:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251537.946708:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c039c0 00000400:00000200:7.0:1547251537.946711:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1785 [1] + 0 00000400:00000200:7.0:1547251537.946713:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040b440 00000100:00000200:7.0:1547251537.946715:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:7.0:1547251537.946724:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b440 00000400:00000200:7.0:1547251537.946725:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251537.946727:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df1500 x1622384012442048/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251537.946732:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:7.0:1547251537.946733:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251537.946734:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251537.946755:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442048 00010000:00000200:5.0:1547251537.946764:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11b850 x1622384012442048/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:598/0 lens 224/368 e 0 to 0 dl 1547251543 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251537.946771:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442048, offset 192 00000400:00000200:5.0:1547251537.946773:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251537.946776:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251537.946779:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c039c0 00000400:00000200:5.0:1547251537.946782:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae7e1 [1] + 192 00000100:00000200:5.0:1547251537.946785:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df1500 x1622384012442048/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251537.946794:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251537.946796:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad6e8 00000400:00000200:5.0:1547251537.946797:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251537.946798:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251537.946802:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df1500 x1622384012442048/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251537.946803:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251537.946807:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df1500 x1622384012442048/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251537.946811:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00010000:00080000:7.0:1547251537.946814:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df1500 x1622384012442048/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251544 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251539.962674:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442064, portal 4 00000100:00000200:8.0:1547251539.962677:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442064, offset 0 00000400:00000200:8.0:1547251539.962680:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:8.0:1547251539.962682:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:8.0:1547251539.962686:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22969] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251539.962689:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22969] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22969] 00000400:00000200:8.0:1547251539.962691:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251539.962693:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251539.962694:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251539.962695:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9716, 9717] 00000400:00000200:8.0:1547251539.962697:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:8.0:1547251539.962701:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251539.962704:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000100:00000200:15.0:1547251539.962706:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442080, portal 4 00000800:00000200:8.0:1547251539.962707:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251539.962708:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:8.0:1547251539.962709:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:15.0:1547251539.962710:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442080, offset 0 00000800:00000200:8.0:1547251539.962712:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:15.0:1547251539.962713:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:15.0:1547251539.962716:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:15.0:1547251539.962719:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22970] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251539.962722:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22969] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22970] 00000400:00000200:15.0:1547251539.962724:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:15.0:1547251539.962726:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251539.962727:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251539.962729:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9717, 9717] 00000400:00000200:15.0:1547251539.962731:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:15.0:1547251539.962735:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251539.962738:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:15.0:1547251539.962741:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251539.962742:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:15.0:1547251539.962743:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251539.962746:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:7.2:1547251539.962761:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:9.2:1547251539.962770:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251539.962787:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000100:00000200:0.0:1547251539.962789:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442096, portal 4 00000800:00000200:13.0:1547251539.962791:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251539.962791:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:13.0:1547251539.962793:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:0.0:1547251539.962793:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442096, offset 0 00000100:00000200:12.0:1547251539.962795:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cad700 x1622384012442080/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251546 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:13.0:1547251539.962796:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824750900 x1622384012442064/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251546 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251539.962797:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:0.0:1547251539.962800:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:13.0:1547251539.962803:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00000400:00000200:12.0:1547251539.962803:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1aa0 00000400:00000200:0.0:1547251539.962803:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22970] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:13.0:1547251539.962804:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251539.962804:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:0.0:1547251539.962806:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22970] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22970] 00000400:00000200:13.0:1547251539.962808:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:12.0:1547251539.962808:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:0.0:1547251539.962808:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:0.0:1547251539.962810:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:12.0:1547251539.962811:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:0.0:1547251539.962811:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:13.0:1547251539.962812:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:0.0:1547251539.962812:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9717, 9718] 00000800:00000200:12.0:1547251539.962813:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:13.0:1547251539.962814:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000400:00000200:0.0:1547251539.962814:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:0.0:1547251539.962819:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251539.962822:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:0.0:1547251539.962825:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251539.962826:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:0.0:1547251539.962827:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:0.0:1547251539.962830:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:7.2:1547251539.962843:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251539.962855:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251539.962863:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:12.0:1547251539.962867:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f7b00 x1622384012442096/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251546 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251539.962869:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:12.0:1547251539.962873:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665d48 00000400:00000200:12.0:1547251539.962875:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251539.962878:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251539.962880:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:9.2:1547251539.962882:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:13.0:1547251539.962890:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:7.2:1547251539.962895:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:13.0:1547251539.962897:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000800:00000200:12.0:1547251539.962898:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251539.962900:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000400:00000200:13.0:1547251539.962902:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251539.962902:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:13.0:1547251539.962907:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c039d0 00000400:00000200:12.0:1547251539.962907:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251539.962911:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f5bd [1] + 192 00000400:00000200:12.0:1547251539.962911:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c039e0 00000400:00000200:12.0:1547251539.962915:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f5cd [1] + 192 00000100:00000200:13.0:1547251539.962920:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824750900 x1622384012442064/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251546 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251539.962929:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:12.0:1547251539.962931:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cad700 x1622384012442080/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251546 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251539.962932:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:13.0:1547251539.962933:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:13.0:1547251539.962935:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:9.2:1547251539.962945:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:13.0:1547251539.962946:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:13.0:1547251539.962948:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:13.0:1547251539.962950:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251539.962951:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251539.962953:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (21)++ 00000100:00000200:8.0:1547251539.962954:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824750900 x1622384012442064/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251546 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251539.962955:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251539.962955:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251539.962956:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000400:00000200:13.0:1547251539.962958:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c039f0 00000800:00000200:12.0:1547251539.962958:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:8.0:1547251539.962962:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824750900 x1622384012442064/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251546 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251539.962968:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688000 00000100:00000200:15.0:1547251539.962971:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cad700 x1622384012442080/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251546 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251539.962973:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824750900 x1622384012442064/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251546 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:15.0:1547251539.962977:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cad700 x1622384012442080/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251546 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251539.962981:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00000400:00000200:13.0:1547251539.962984:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f5ed [1] + 192 00010000:00080000:15.0:1547251539.962985:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cad700 x1622384012442080/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251546 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:13.0:1547251539.962988:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f7b00 x1622384012442096/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251546 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251539.962997:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251539.962998:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:13.0:1547251539.962999:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:13.0:1547251539.963000:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000100:00000200:0.0:1547251539.963007:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f7b00 x1622384012442096/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251546 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251539.963012:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f7b00 x1622384012442096/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251546 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251539.963022:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00010000:00080000:0.0:1547251539.963025:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f7b00 x1622384012442096/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251546 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251542.074683:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442112, portal 4 00000100:00000200:15.0:1547251542.074685:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442128, portal 4 00000100:00000200:9.0:1547251542.074686:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442112, offset 0 00000100:00000200:15.0:1547251542.074688:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442128, offset 0 00000400:00000200:9.0:1547251542.074689:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:15.0:1547251542.074692:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251542.074692:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251542.074694:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22971] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251542.074695:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:15.0:1547251542.074698:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22971] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251542.074698:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22970] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22971] 00000400:00000200:9.0:1547251542.074700:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:15.0:1547251542.074701:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22970] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22971] 00000400:00000200:9.0:1547251542.074701:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251542.074703:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251542.074703:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251542.074704:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9723, 9722] 00000400:00000200:15.0:1547251542.074705:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251542.074706:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251542.074706:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:15.0:1547251542.074708:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9723, 9723] 00000400:00000200:15.0:1547251542.074710:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:9.0:1547251542.074711:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251542.074713:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:15.0:1547251542.074716:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251542.074719:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:9.0:1547251542.074720:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251542.074722:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251542.074722:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:15.0:1547251542.074723:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:9.0:1547251542.074723:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:15.0:1547251542.074725:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:9.0:1547251542.074726:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:15.0:1547251542.074728:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000100:00000200:2.0:1547251542.074751:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442144, portal 4 00000100:00000200:2.0:1547251542.074754:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442144, offset 0 00000400:00000200:2.0:1547251542.074759:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:2.0:1547251542.074762:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:2.0:1547251542.074765:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22972] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251542.074768:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22971] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22972] 00000400:00000200:2.0:1547251542.074771:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251542.074772:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251542.074773:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:7.2:1547251542.074775:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:2.0:1547251542.074775:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 6], s:[9723, 9724] 00000400:00000200:2.0:1547251542.074776:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:2.0:1547251542.074781:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251542.074784:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:2.0:1547251542.074787:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251542.074788:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:2.0:1547251542.074789:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:5.2:1547251542.074791:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:2.0:1547251542.074792:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000800:00000200:11.0:1547251542.074800:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:7.2:1547251542.074804:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251542.074805:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000100:00000200:11.0:1547251542.074810:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf5d00 x1622384012442112/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251542.074817:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea2a8 00000400:00000200:11.0:1547251542.074818:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251542.074818:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251542.074821:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000400:00000200:11.0:1547251542.074822:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251542.074825:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000100:00000200:10.0:1547251542.074825:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cad100 x1622384012442128/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251542.074827:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:11.0:1547251542.074829:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251542.074830:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000400:00000200:10.0:1547251542.074831:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1ee0 00000400:00000200:10.0:1547251542.074832:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:11.0:1547251542.074833:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328db900 x1622384012442144/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251542.074836:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251542.074838:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:11.0:1547251542.074839:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a4c8 00000400:00000200:11.0:1547251542.074840:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251542.074840:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:10.0:1547251542.074841:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:11.0:1547251542.074843:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251542.074846:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:7.2:1547251542.074891:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:5.2:1547251542.074900:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:10.0:1547251542.074908:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251542.074911:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251542.074914:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251542.074916:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251542.074916:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251542.074921:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251542.074921:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03a00 00000400:00000200:11.0:1547251542.074925:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03a20 00000400:00000200:10.0:1547251542.074925:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6429 [1] + 192 00000400:00000200:11.0:1547251542.074929:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6459 [1] + 192 00000100:00000200:10.0:1547251542.074933:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf5d00 x1622384012442112/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251542.074954:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251542.074955:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328db900 x1622384012442144/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251542.074956:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251542.074958:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251542.074959:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251542.074961:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251542.074963:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251542.074966:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251542.074968:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000100:00000200:9.0:1547251542.074968:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf5d00 x1622384012442112/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251542.074969:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000400:00000200:10.0:1547251542.074969:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251542.074971:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251542.074972:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000400:00000200:10.0:1547251542.074972:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03a10 00000800:00000200:11.0:1547251542.074974:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000100:00000200:9.0:1547251542.074974:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf5d00 x1622384012442112/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251542.074978:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaa18 00010000:00080000:9.0:1547251542.074981:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf5d00 x1622384012442112/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:2.0:1547251542.074981:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328db900 x1622384012442144/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251542.074986:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328db900 x1622384012442144/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251542.074991:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00000400:00000200:10.0:1547251542.074994:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6439 [1] + 192 00010000:00080000:2.0:1547251542.074994:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328db900 x1622384012442144/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251542.074998:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cad100 x1622384012442128/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251542.075007:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251542.075008:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251542.075009:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251542.075010:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:15.0:1547251542.075055:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cad100 x1622384012442128/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:15.0:1547251542.075061:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cad100 x1622384012442128/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251542.075065:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00010000:00080000:15.0:1547251542.075069:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cad100 x1622384012442128/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251542.538685:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442160, portal 4 00000100:00000200:8.0:1547251542.538688:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442160, offset 0 00000400:00000200:8.0:1547251542.538691:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251542.538694:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251542.538696:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22972] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251542.538699:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22972] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22972] 00000400:00000200:8.0:1547251542.538701:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251542.538703:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251542.538704:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251542.538705:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9716, 9716] 00000400:00000200:8.0:1547251542.538708:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:8.0:1547251542.538713:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251542.538716:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:8.0:1547251542.538719:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251542.538720:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:8.0:1547251542.538721:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000100:00000200:0.0:1547251542.538721:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442176, portal 4 00000800:00000200:8.0:1547251542.538724:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000100:00000200:0.0:1547251542.538724:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442176, offset 0 00000400:00000200:0.0:1547251542.538727:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:0.0:1547251542.538730:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:0.0:1547251542.538732:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22973] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251542.538736:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22972] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22973] 00000100:00000200:13.0:1547251542.538738:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442192, portal 4 00000400:00000200:0.0:1547251542.538738:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:12.2:1547251542.538739:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000400:00000200:0.0:1547251542.538739:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251542.538741:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:13.0:1547251542.538742:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442192, offset 0 00000400:00000200:0.0:1547251542.538742:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9716, 9717] 00000400:00000200:0.0:1547251542.538744:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:13.0:1547251542.538746:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:13.0:1547251542.538749:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:0.0:1547251542.538749:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251542.538752:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:13.0:1547251542.538753:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22973] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:0.0:1547251542.538755:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251542.538756:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000400:00000200:13.0:1547251542.538757:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22973] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22973] 00000800:00000200:0.0:1547251542.538757:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:13.0:1547251542.538760:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:0.0:1547251542.538760:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:13.0:1547251542.538762:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251542.538766:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251542.538767:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9717, 9717] 00000800:00000200:0.2:1547251542.538768:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:13.0:1547251542.538770:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:13.0:1547251542.538777:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251542.538780:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:13.0:1547251542.538783:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251542.538785:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (22)++ 00000800:00000200:13.0:1547251542.538786:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00000800:00000200:13.0:1547251542.538789:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (24)-- 00000800:00000200:4.0:1547251542.538796:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:4.0:1547251542.538799:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000100:00000200:4.0:1547251542.538803:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824752d00 x1622384012442160/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251542.538808:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688440 00000400:00000200:4.0:1547251542.538809:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251542.538812:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251542.538814:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251542.538816:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251542.538817:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:4.0:1547251542.538819:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f7800 x1622384012442176/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:7.0:1547251542.538821:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:7.0:1547251542.538824:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000100:00000200:7.0:1547251542.538828:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8caa400 x1622384012442192/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251542.538828:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665dd0 00000400:00000200:4.0:1547251542.538829:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251542.538831:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251542.538832:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:7.0:1547251542.538833:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b990 00000400:00000200:7.0:1547251542.538834:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251542.538834:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:4.0:1547251542.538835:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:7.0:1547251542.538838:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:7.0:1547251542.538840:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:0.2:1547251542.538867:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:4.0:1547251542.538880:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:4.0:1547251542.538889:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251542.538890:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251542.538892:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251542.538892:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251542.538893:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:0.2:1547251542.538901:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:4.0:1547251542.538907:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251542.538912:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251542.538915:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:7.0:1547251542.538919:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:4.0:1547251542.538919:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03a30 00000800:00000200:0.2:1547251542.538920:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000400:00000200:4.0:1547251542.538922:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3155 [1] + 192 00000800:00000200:12.2:1547251542.538926:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000100:00000200:4.0:1547251542.538926:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824752d00 x1622384012442160/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:7.0:1547251542.538928:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:7.0:1547251542.538930:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251542.538933:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251542.538934:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251542.538935:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251542.538945:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:7.0:1547251542.538946:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03a40 00000800:00000200:4.0:1547251542.538946:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251542.538947:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000400:00000200:7.0:1547251542.538949:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3165 [1] + 192 00000800:00000200:4.0:1547251542.538949:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251542.538952:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000100:00000200:8.0:1547251542.538954:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824752d00 x1622384012442160/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251542.538955:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03a50 00000100:00000200:8.0:1547251542.538960:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824752d00 x1622384012442160/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251542.538964:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688000 00000100:00000200:7.0:1547251542.538967:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f7800 x1622384012442176/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251542.538968:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824752d00 x1622384012442160/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:7.0:1547251542.538977:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:7.0:1547251542.538979:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000400:00000200:4.0:1547251542.538979:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3175 [1] + 192 00000800:00000200:7.0:1547251542.538981:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:4.0:1547251542.538981:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8caa400 x1622384012442192/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:7.0:1547251542.538982:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:7.0:1547251542.538983:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:7.0:1547251542.538985:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:4.0:1547251542.538989:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251542.538991:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000100:00000200:0.0:1547251542.538991:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f7800 x1622384012442176/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251542.538992:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251542.538993:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:0.0:1547251542.538997:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f7800 x1622384012442176/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251542.539001:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00000100:00000200:13.0:1547251542.539004:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8caa400 x1622384012442192/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251542.539005:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f7800 x1622384012442176/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:13.0:1547251542.539010:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8caa400 x1622384012442192/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251542.539014:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792bd48 00010000:00080000:13.0:1547251542.539018:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8caa400 x1622384012442192/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251542.938708:0:12678:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442208, portal 4 00000100:00000200:7.0:1547251542.938709:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442224, portal 4 00000100:00000200:10.0:1547251542.938711:0:12678:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442208, offset 0 00000100:00000200:7.0:1547251542.938712:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442224, offset 0 00000400:00000200:10.0:1547251542.938713:0:12678:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251542.938715:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:10.0:1547251542.938716:0:12678:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251542.938718:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:10.0:1547251542.938719:0:12678:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03a60 00000400:00000200:10.0:1547251542.938721:0:12678:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd3e29 [1] + 0 00000400:00000200:7.0:1547251542.938721:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03a70 00000400:00000200:10.0:1547251542.938723:0:12678:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ae6a914198 00000400:00000200:7.0:1547251542.938723:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe18c5 [1] + 0 00000100:00000200:10.0:1547251542.938725:0:12678:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:7.0:1547251542.938725:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040b6e8 00000100:00000200:7.0:1547251542.938727:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:10.0:1547251542.938733:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914198 00000400:00000200:10.0:1547251542.938734:0:12678:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251542.938735:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b6e8 00000100:00000200:10.0:1547251542.938736:0:12678:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960edd00 x1622384012442208/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251542.938736:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251542.938739:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df3000 x1622384012442224/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251542.938741:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a9144c8 00000400:00000200:10.0:1547251542.938742:0:12678:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251542.938743:0:12678:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251542.938743:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:7.0:1547251542.938744:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251542.938745:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251542.938754:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442208 00010000:00000200:8.0:1547251542.938763:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3dec50 x1622384012442208/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:603/0 lens 224/368 e 0 to 0 dl 1547251548 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251542.938768:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442224 00000100:00000200:8.0:1547251542.938769:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442208, offset 192 00000400:00000200:8.0:1547251542.938772:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251542.938774:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251542.938777:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03a60 00010000:00000200:5.0:1547251542.938777:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11b450 x1622384012442224/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:603/0 lens 224/368 e 0 to 0 dl 1547251548 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:8.0:1547251542.938780:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae821 [1] + 192 00000100:00000200:8.0:1547251542.938783:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960edd00 x1622384012442208/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251542.938784:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442224, offset 192 00000400:00000200:5.0:1547251542.938787:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251542.938790:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251542.938792:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251542.938793:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03a70 00000400:00000200:8.0:1547251542.938794:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688000 00000400:00000200:8.0:1547251542.938795:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251542.938796:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:5.0:1547251542.938796:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae831 [1] + 192 00000100:00000200:5.0:1547251542.938799:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df3000 x1622384012442224/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251542.938801:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:5.0:1547251542.938809:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251542.938810:0:12678:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960edd00 x1622384012442208/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251542.938811:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073addd0 00000400:00000200:5.0:1547251542.938812:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251542.938813:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:10.0:1547251542.938815:0:12678:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960edd00 x1622384012442208/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251542.938818:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:10.0:1547251542.938819:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914550 00000100:00000200:7.0:1547251542.938821:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df3000 x1622384012442224/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:10.0:1547251542.938822:0:12678:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba960edd00 x1622384012442208/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251542.938830:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df3000 x1622384012442224/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251542.938834:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00010000:00080000:7.0:1547251542.938837:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df3000 x1622384012442224/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251542.954689:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442240, portal 4 00000100:00000200:6.0:1547251542.954692:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442240, offset 0 00000400:00000200:6.0:1547251542.954695:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251542.954698:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251542.954701:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03a80 00000400:00000200:6.0:1547251542.954703:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe18f5 [1] + 0 00000400:00000200:6.0:1547251542.954705:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040b198 00000100:00000200:6.0:1547251542.954707:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251542.954716:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b198 00000400:00000200:6.0:1547251542.954716:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251542.954719:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37932100 x1622384012442240/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251542.954724:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990bb0 00000400:00000200:6.0:1547251542.954724:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251542.954725:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251542.954748:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442240 00010000:00000200:4.0:1547251542.954757:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11b050 x1622384012442240/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:603/0 lens 224/368 e 0 to 0 dl 1547251548 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251542.954764:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442240, offset 192 00000400:00000200:4.0:1547251542.954766:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251542.954769:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251542.954772:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03a80 00000400:00000200:4.0:1547251542.954775:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae841 [1] + 192 00000100:00000200:4.0:1547251542.954778:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37932100 x1622384012442240/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251542.954787:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251542.954789:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b770 00000400:00000200:4.0:1547251542.954790:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251542.954791:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251542.954794:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37932100 x1622384012442240/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251542.954796:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251542.954798:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37932100 x1622384012442240/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251542.954804:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00010000:00080000:6.0:1547251542.954807:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37932100 x1622384012442240/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251549 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251544.970683:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442272, portal 4 00000100:00000200:9.0:1547251544.970686:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442272, offset 0 00000100:00000200:1.0:1547251544.970688:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442256, portal 4 00000400:00000200:9.0:1547251544.970689:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000100:00000200:1.0:1547251544.970691:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442256, offset 0 00000400:00000200:9.0:1547251544.970692:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:1.0:1547251544.970694:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:9.0:1547251544.970695:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22974] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251544.970697:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:9.0:1547251544.970698:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22973] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22974] 00000400:00000200:1.0:1547251544.970699:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22974] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251544.970701:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251544.970702:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251544.970702:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22974] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22974] 00000400:00000200:9.0:1547251544.970704:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251544.970705:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9718, 9718] 00000400:00000200:1.0:1547251544.970705:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251544.970707:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:1.0:1547251544.970707:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251544.970709:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251544.970711:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9718, 9719] 00000400:00000200:9.0:1547251544.970712:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000400:00000200:1.0:1547251544.970713:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000800:00000200:9.0:1547251544.970714:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:9.0:1547251544.970717:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000400:00000200:1.0:1547251544.970717:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251544.970719:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:9.0:1547251544.970720:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:1.0:1547251544.970720:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:9.0:1547251544.970723:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:1.0:1547251544.970723:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251544.970725:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:1.0:1547251544.970726:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:1.0:1547251544.970728:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:12.0:1547251544.970731:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442288, portal 4 00000800:00000200:9.2:1547251544.970732:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000100:00000200:12.0:1547251544.970735:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442288, offset 0 00000400:00000200:12.0:1547251544.970739:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:12.0:1547251544.970743:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:12.0:1547251544.970745:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22975] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251544.970748:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22974] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22975] 00000400:00000200:12.0:1547251544.970750:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:12.0:1547251544.970752:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251544.970753:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251544.970755:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9719, 9719] 00000400:00000200:12.0:1547251544.970757:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251544.970761:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251544.970764:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:12.0:1547251544.970767:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251544.970769:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251544.970770:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:12.0:1547251544.970773:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (24)-- 00000800:00000200:7.2:1547251544.970778:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251544.970789:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:12.0:1547251544.970791:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (24)-- 00000100:00000200:12.0:1547251544.970795:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf4200 x1622384012442272/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251551 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251544.970798:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:13.0:1547251544.970800:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000400:00000200:12.0:1547251544.970802:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea000 00000100:00000200:13.0:1547251544.970804:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b4b00 x1622384012442256/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251551 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251544.970804:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251544.970807:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:13.0:1547251544.970810:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e25d8 00000800:00000200:12.0:1547251544.970810:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)-- 00000400:00000200:13.0:1547251544.970811:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:12.0:1547251544.970811:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251544.970813:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000400:00000200:13.0:1547251544.970814:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000100:00000200:12.0:1547251544.970815:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a6900 x1622384012442288/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251551 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251544.970817:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:13.0:1547251544.970819:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:13.0:1547251544.970820:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:12.0:1547251544.970821:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038198 00000400:00000200:12.0:1547251544.970822:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251544.970824:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251544.970827:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:9.2:1547251544.970869:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:13.0:1547251544.970881:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:13.0:1547251544.970891:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.0.11.93@o2ib10 00000800:00000200:13.0:1547251544.970893:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:13.0:1547251544.970895:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:13.0:1547251544.970896:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251544.970897:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:7.2:1547251544.970899:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251544.970906:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:13.0:1547251544.970911:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:13.0:1547251544.970914:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251544.970914:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251544.970916:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:13.0:1547251544.970920:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251544.970921:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251544.970925:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03aa0 00000400:00000200:12.0:1547251544.970925:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03a90 00000400:00000200:12.0:1547251544.970929:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f62d [1] + 192 00000400:00000200:13.0:1547251544.970933:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f61d [1] + 192 00000100:00000200:12.0:1547251544.970946:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b4b00 x1622384012442256/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251551 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251544.970958:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:13.0:1547251544.970959:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf4200 x1622384012442272/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251551 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251544.970960:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251544.970962:0:12665:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (23)++ 00000800:00000200:12.0:1547251544.970964:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (24)-- 00000800:00000200:12.0:1547251544.970966:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251544.970968:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000400:00000200:13.0:1547251544.970969:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251544.970970:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000800:00000200:13.0:1547251544.970971:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (23)++ 00000800:00000200:13.0:1547251544.970973:0:12666:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (24)++ 00000400:00000200:12.0:1547251544.970974:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000100:00000200:1.0:1547251544.970974:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b4b00 x1622384012442256/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251551 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251544.970975:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (25)-- 00000800:00000200:13.0:1547251544.970976:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (24)-- 00000800:00000200:13.0:1547251544.970978:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000400:00000200:12.0:1547251544.970978:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03ab0 00000800:00000200:13.0:1547251544.970979:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:13.0:1547251544.970980:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:13.0:1547251544.970981:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (23)++ 00000100:00000200:1.0:1547251544.970981:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b4b00 x1622384012442256/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251551 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251544.970982:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (24)-- 00000800:00000200:13.0:1547251544.970983:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (23)-- 00000800:00000200:13.0:1547251544.970984:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:13.0:1547251544.970986:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000400:00000200:1.0:1547251544.970987:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00000100:00000200:9.0:1547251544.970990:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf4200 x1622384012442272/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251551 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:1.0:1547251544.970991:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b4b00 x1622384012442256/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251551 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251544.970996:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf4200 x1622384012442272/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251551 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251544.971000:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaa18 00000400:00000200:12.0:1547251544.971003:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f63d [1] + 192 00010000:00080000:9.0:1547251544.971003:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf4200 x1622384012442272/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251551 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:12.0:1547251544.971007:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a6900 x1622384012442288/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251551 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251544.971018:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251544.971020:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251544.971021:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251544.971022:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:14.0:1547251544.971067:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a6900 x1622384012442288/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251551 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:14.0:1547251544.971073:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a6900 x1622384012442288/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251551 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251544.971078:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038880 00010000:00080000:14.0:1547251544.971082:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a6900 x1622384012442288/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251551 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251547.082682:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442304, portal 4 00000100:00000200:8.0:1547251547.082685:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442304, offset 0 00000400:00000200:8.0:1547251547.082688:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251547.082692:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251547.082694:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22975] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251547.082698:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22975] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22975] 00000100:00000200:3.0:1547251547.082699:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442320, portal 4 00000400:00000200:8.0:1547251547.082700:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251547.082702:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:3.0:1547251547.082702:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442320, offset 0 00000400:00000200:8.0:1547251547.082703:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251547.082704:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9724, 9724] 00000400:00000200:3.0:1547251547.082705:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251547.082706:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:3.0:1547251547.082708:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:3.0:1547251547.082710:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22976] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251547.082711:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000400:00000200:3.0:1547251547.082713:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22975] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22976] 00000800:00000200:8.0:1547251547.082714:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:8.0:1547251547.082716:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000400:00000200:3.0:1547251547.082716:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:3.0:1547251547.082717:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251547.082718:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:8.0:1547251547.082719:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000400:00000200:3.0:1547251547.082719:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251547.082720:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9724, 9725] 00000800:00000200:8.0:1547251547.082722:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000400:00000200:3.0:1547251547.082722:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:3.0:1547251547.082727:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251547.082729:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000100:00000200:12.0:1547251547.082732:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442336, portal 4 00000800:00000200:3.0:1547251547.082732:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251547.082733:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:3.0:1547251547.082735:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000100:00000200:12.0:1547251547.082736:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442336, offset 0 00000800:00000200:3.0:1547251547.082737:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000400:00000200:12.0:1547251547.082741:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:12.0:1547251547.082744:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:12.0:1547251547.082746:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22976] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251547.082749:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22976] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22976] 00000400:00000200:12.0:1547251547.082753:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:12.0:1547251547.082754:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251547.082756:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251547.082757:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9725, 9725] 00000400:00000200:12.0:1547251547.082759:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:12.0:1547251547.082764:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251547.082767:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:12.0:1547251547.082770:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:5.2:1547251547.082771:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:12.0:1547251547.082772:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (21)++ 00000800:00000200:12.0:1547251547.082773:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (22)++ 00000800:00000200:12.0:1547251547.082776:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (24)-- 00000800:00000200:11.0:1547251547.082788:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (23)++ 00000800:00000200:7.2:1547251547.082797:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251547.082798:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (24)-- 00000100:00000200:11.0:1547251547.082802:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824755100 x1622384012442304/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251547.082810:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688440 00000400:00000200:11.0:1547251547.082812:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251547.082815:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251547.082815:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (23)++ 00000800:00000200:11.0:1547251547.082818:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (24)-- 00000800:00000200:10.0:1547251547.082819:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (24)-- 00000800:00000200:11.0:1547251547.082820:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251547.082821:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000100:00000200:10.0:1547251547.082822:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a7b00 x1622384012442336/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251547.082824:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c5700 x1622384012442320/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251547.082828:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038440 00000400:00000200:11.0:1547251547.082830:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555550 00000400:00000200:10.0:1547251547.082830:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251547.082832:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251547.082833:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251547.082835:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251547.082835:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251547.082837:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251547.082838:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:11.0:1547251547.082839:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:7.2:1547251547.082903:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:5.2:1547251547.082909:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:5.2:1547251547.082914:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:10.0:1547251547.082917:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251547.082921:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251547.082922:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251547.082924:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251547.082926:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251547.082929:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251547.082931:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03ac0 00000400:00000200:11.0:1547251547.082933:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03ad0 00000400:00000200:10.0:1547251547.082935:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd64b9 [1] + 192 00000400:00000200:11.0:1547251547.082948:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd64d9 [1] + 192 00000100:00000200:10.0:1547251547.082952:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824755100 x1622384012442304/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251547.082964:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251547.082965:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c5700 x1622384012442320/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251547.082966:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251547.082968:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251547.082970:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251547.082972:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251547.082974:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251547.082976:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251547.082978:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000400:00000200:10.0:1547251547.082978:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000100:00000200:8.0:1547251547.082978:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824755100 x1622384012442304/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251547.082979:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251547.082981:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251547.082982:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000400:00000200:10.0:1547251547.082982:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03ae0 00000800:00000200:11.0:1547251547.082984:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:8.0:1547251547.082984:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824755100 x1622384012442304/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251547.082985:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:8.0:1547251547.082988:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688220 00010000:00080000:8.0:1547251547.082991:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824755100 x1622384012442304/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251547.082991:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c5700 x1622384012442320/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251547.082996:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c5700 x1622384012442320/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251547.083005:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00000400:00000200:10.0:1547251547.083008:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd64e9 [1] + 192 00010000:00080000:3.0:1547251547.083009:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c5700 x1622384012442320/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251547.083012:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a7b00 x1622384012442336/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251547.083021:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251547.083023:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251547.083024:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251547.083024:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:12.0:1547251547.083032:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a7b00 x1622384012442336/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251547.083037:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a7b00 x1622384012442336/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251547.083042:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038198 00010000:00080000:12.0:1547251547.083045:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a7b00 x1622384012442336/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:1.0:1547251547.546686:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442352, portal 4 00000100:00000200:1.0:1547251547.546688:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442352, offset 0 00000400:00000200:1.0:1547251547.546692:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:1.0:1547251547.546694:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:1.0:1547251547.546697:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22977] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251547.546700:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22976] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22977] 00000400:00000200:1.0:1547251547.546702:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000100:00000200:12.0:1547251547.546704:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442368, portal 4 00000400:00000200:1.0:1547251547.546704:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251547.546705:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:12.0:1547251547.546707:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442368, offset 0 00000400:00000200:1.0:1547251547.546707:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9717, 9718] 00000400:00000200:1.0:1547251547.546708:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000100:00000200:9.0:1547251547.546710:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442384, portal 4 00000400:00000200:12.0:1547251547.546711:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000100:00000200:9.0:1547251547.546713:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442384, offset 0 00000400:00000200:1.0:1547251547.546713:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:12.0:1547251547.546714:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:12.0:1547251547.546716:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22977] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251547.546716:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:1.0:1547251547.546716:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:12.0:1547251547.546719:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22977] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22977] 00000400:00000200:9.0:1547251547.546719:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:1.0:1547251547.546719:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251547.546721:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000400:00000200:12.0:1547251547.546722:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251547.546722:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22977] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:1.0:1547251547.546722:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:12.0:1547251547.546723:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251547.546725:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251547.546725:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22977] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22978] 00000800:00000200:1.0:1547251547.546725:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:12.0:1547251547.546726:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9718, 9718] 00000400:00000200:9.0:1547251547.546727:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:12.0:1547251547.546728:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:9.0:1547251547.546729:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251547.546730:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251547.546732:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9718, 9719] 00000400:00000200:12.0:1547251547.546733:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:9.0:1547251547.546735:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:12.0:1547251547.546738:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:12.0:1547251547.546741:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:9.0:1547251547.546741:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:0.2:1547251547.546742:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:12.0:1547251547.546743:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:12.0:1547251547.546744:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:9.0:1547251547.546744:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:12.0:1547251547.546747:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:9.0:1547251547.546747:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251547.546749:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:9.0:1547251547.546750:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:9.0:1547251547.546753:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:12.2:1547251547.546755:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:12.2:1547251547.546763:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000800:00000200:7.0:1547251547.546797:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:7.0:1547251547.546800:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:7.0:1547251547.546803:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b4e00 x1622384012442352/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251547.546808:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2198 00000400:00000200:7.0:1547251547.546810:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251547.546812:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251547.546812:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:7.0:1547251547.546815:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251547.546815:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:7.0:1547251547.546817:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:7.0:1547251547.546818:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000100:00000200:4.0:1547251547.546819:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b2dd100 x1622384012442368/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251547.546820:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf3900 x1622384012442384/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251547.546824:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038088 00000400:00000200:7.0:1547251547.546825:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea660 00000400:00000200:7.0:1547251547.546825:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251547.546825:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251547.546828:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251547.546828:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:7.0:1547251547.546829:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251547.546831:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:7.0:1547251547.546832:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:4.0:1547251547.546833:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:7.0:1547251547.546834:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:0.2:1547251547.546902:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251547.546907:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:4.0:1547251547.546915:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:4.0:1547251547.546918:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000800:00000200:12.2:1547251547.546919:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:7.0:1547251547.546921:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000400:00000200:4.0:1547251547.546921:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:7.0:1547251547.546924:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251547.546925:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03af0 00000400:00000200:7.0:1547251547.546927:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251547.546928:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe31e5 [1] + 192 00000400:00000200:7.0:1547251547.546931:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03b10 00000100:00000200:4.0:1547251547.546932:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b4e00 x1622384012442352/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251547.546953:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251547.546954:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3205 [1] + 192 00000800:00000200:4.0:1547251547.546955:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251547.546956:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251547.546957:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:7.0:1547251547.546958:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf3900 x1622384012442384/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251547.546959:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:4.0:1547251547.546961:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251547.546963:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251547.546966:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:7.0:1547251547.546968:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251547.546969:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03b00 00000800:00000200:7.0:1547251547.546971:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000100:00000200:1.0:1547251547.546971:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b4e00 x1622384012442352/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:7.0:1547251547.546973:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:7.0:1547251547.546974:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:7.0:1547251547.546975:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:7.0:1547251547.546976:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:1.0:1547251547.546978:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b4e00 x1622384012442352/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251547.546984:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00000400:00000200:4.0:1547251547.546987:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe31f5 [1] + 192 00010000:00080000:1.0:1547251547.546988:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b4e00 x1622384012442352/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251547.546990:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf3900 x1622384012442384/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251547.546996:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf3900 x1622384012442384/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251547.547000:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaa18 00000100:00000200:4.0:1547251547.547003:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b2dd100 x1622384012442368/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251547.547004:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf3900 x1622384012442384/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:4.0:1547251547.547011:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251547.547013:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251547.547014:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251547.547015:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:12.0:1547251547.547020:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b2dd100 x1622384012442368/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251547.547026:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b2dd100 x1622384012442368/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251547.547030:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038198 00010000:00080000:12.0:1547251547.547034:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b17b2dd100 x1622384012442368/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251547.946680:0:12678:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442400, portal 4 00000100:00000200:11.0:1547251547.946682:0:12678:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442400, offset 0 00000400:00000200:11.0:1547251547.946685:0:12678:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:11.0:1547251547.946688:0:12678:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:11.0:1547251547.946690:0:12678:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03b20 00000400:00000200:11.0:1547251547.946693:0:12678:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd3e99 [1] + 0 00000400:00000200:11.0:1547251547.946695:0:12678:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ba9d367b28 00000100:00000200:11.0:1547251547.946697:0:12678:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:11.0:1547251547.946705:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba9d367b28 00000400:00000200:11.0:1547251547.946706:0:12678:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251547.946708:0:12678:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07d100 x1622384012442400/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251547.946713:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0baa0 00000400:00000200:11.0:1547251547.946713:0:12678:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251547.946714:0:12678:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251547.946718:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442416, portal 4 00000100:00000200:6.0:1547251547.946721:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442416, offset 0 00000400:00000200:6.0:1547251547.946724:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251547.946727:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251547.946730:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03b30 00000400:00000200:6.0:1547251547.946732:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1975 [1] + 0 00000100:00000200:8.0:1547251547.946735:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442400 00000400:00000200:6.0:1547251547.946735:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b6459904c8 00000100:00000200:6.0:1547251547.946737:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00010000:00000200:8.0:1547251547.946743:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3de850 x1622384012442400/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:608/0 lens 224/368 e 0 to 0 dl 1547251553 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:6.0:1547251547.946744:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459904c8 00000400:00000200:6.0:1547251547.946745:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251547.946748:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37933000 x1622384012442416/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251547.946750:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442400, offset 192 00000400:00000200:8.0:1547251547.946752:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251547.946752:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990bb0 00000400:00000200:6.0:1547251547.946753:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251547.946754:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:8.0:1547251547.946755:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251547.946758:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03b20 00000400:00000200:8.0:1547251547.946761:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae881 [1] + 192 00000100:00000200:8.0:1547251547.946764:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07d100 x1622384012442400/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251547.946775:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442416 00000400:00000200:8.0:1547251547.946777:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251547.946779:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688220 00000400:00000200:8.0:1547251547.946780:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251547.946782:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:11.0:1547251547.946784:0:12678:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07d100 x1622384012442400/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00000200:4.0:1547251547.946784:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11ac50 x1622384012442416/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:608/0 lens 224/368 e 0 to 0 dl 1547251553 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:8.0:1547251547.946787:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:11.0:1547251547.946789:0:12678:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07d100 x1622384012442400/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251547.946791:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442416, offset 192 00000400:00000200:11.0:1547251547.946793:0:12678:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bb28 00000400:00000200:4.0:1547251547.946793:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00010000:00080000:11.0:1547251547.946796:0:12678:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b17b07d100 x1622384012442400/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:4.0:1547251547.946796:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251547.946799:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03b30 00000400:00000200:4.0:1547251547.946802:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae891 [1] + 192 00000100:00000200:4.0:1547251547.946805:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37933000 x1622384012442416/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251547.946814:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251547.946816:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bf68 00000400:00000200:4.0:1547251547.946817:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251547.946818:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251547.946821:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37933000 x1622384012442416/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251547.946823:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251547.946826:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37933000 x1622384012442416/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251547.946830:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00010000:00080000:6.0:1547251547.946833:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37933000 x1622384012442416/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251547.962703:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442432, portal 4 00000100:00000200:7.0:1547251547.962705:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442432, offset 0 00000400:00000200:7.0:1547251547.962708:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251547.962711:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251547.962713:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03b40 00000400:00000200:7.0:1547251547.962716:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe19a5 [1] + 0 00000400:00000200:7.0:1547251547.962718:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990908 00000100:00000200:7.0:1547251547.962720:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:7.0:1547251547.962729:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990908 00000400:00000200:7.0:1547251547.962731:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251547.962733:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df2d00 x1622384012442432/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251547.962738:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:7.0:1547251547.962739:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251547.962740:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251547.962751:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442432 00010000:00000200:4.0:1547251547.962759:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11a850 x1622384012442432/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:608/0 lens 224/368 e 0 to 0 dl 1547251553 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251547.962766:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442432, offset 192 00000400:00000200:4.0:1547251547.962769:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251547.962772:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251547.962775:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03b40 00000400:00000200:4.0:1547251547.962777:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae8a1 [1] + 192 00000100:00000200:4.0:1547251547.962780:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df2d00 x1622384012442432/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251547.962790:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251547.962792:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bb28 00000400:00000200:4.0:1547251547.962792:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251547.962794:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251547.962798:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df2d00 x1622384012442432/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251547.962799:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251547.962803:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df2d00 x1622384012442432/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251547.962806:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00010000:00080000:7.0:1547251547.962809:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df2d00 x1622384012442432/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251554 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251549.978677:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442448, portal 4 00000100:00000200:8.0:1547251549.978680:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442448, offset 0 00000400:00000200:8.0:1547251549.978683:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:8.0:1547251549.978685:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:8.0:1547251549.978688:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22979] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251549.978691:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22977] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22979] 00000100:00000200:14.0:1547251549.978692:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442464, portal 4 00000400:00000200:8.0:1547251549.978693:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:8.0:1547251549.978694:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:14.0:1547251549.978695:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442464, offset 0 00000400:00000200:8.0:1547251549.978696:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251549.978697:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9719, 9720] 00000400:00000200:14.0:1547251549.978698:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:8.0:1547251549.978699:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251549.978701:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:14.0:1547251549.978703:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22979] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:14.0:1547251549.978707:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22978] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22979] 00000400:00000200:14.0:1547251549.978709:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:14.0:1547251549.978710:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251549.978712:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251549.978713:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9720, 9720] 00000400:00000200:14.0:1547251549.978715:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251549.978720:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251549.978722:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:8.0:1547251549.978725:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251549.978728:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:14.0:1547251549.978729:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251549.978730:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:8.0:1547251549.978731:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251549.978732:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:8.0:1547251549.978733:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:14.0:1547251549.978734:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:8.0:1547251549.978734:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:8.0:1547251549.978736:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:2.0:1547251549.978736:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442480, portal 4 00000100:00000200:2.0:1547251549.978740:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442480, offset 0 00000400:00000200:2.0:1547251549.978744:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:2.0:1547251549.978747:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:2.0:1547251549.978750:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22979] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:9.2:1547251549.978752:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:2.0:1547251549.978753:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:254, d:10, s:22979] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22979] 00000400:00000200:2.0:1547251549.978755:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:2.0:1547251549.978757:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251549.978758:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.2:1547251549.978759:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:2.0:1547251549.978759:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9720, 9721] 00000400:00000200:2.0:1547251549.978761:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:2.0:1547251549.978766:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251549.978768:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:2.0:1547251549.978769:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:2.0:1547251549.978772:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251549.978773:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:2.0:1547251549.978773:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:2.0:1547251549.978774:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:13.0:1547251549.978777:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a5d00 x1622384012442464/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251556 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:2.0:1547251549.978777:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251549.978781:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251549.978783:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000400:00000200:13.0:1547251549.978784:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00000400:00000200:13.0:1547251549.978786:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:12.0:1547251549.978786:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824750000 x1622384012442448/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251556 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251549.978790:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251549.978793:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:12.0:1547251549.978793:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000400:00000200:12.0:1547251549.978794:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:13.0:1547251549.978795:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:13.0:1547251549.978797:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:12.0:1547251549.978798:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251549.978800:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:7.2:1547251549.978826:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:13.0:1547251549.978842:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:13.0:1547251549.978850:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:13.0:1547251549.978854:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328dbc00 x1622384012442480/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251556 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251549.978856:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:13.0:1547251549.978860:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06abb0 00000400:00000200:13.0:1547251549.978861:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251549.978865:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251549.978868:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:7.2:1547251549.978876:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:12.0:1547251549.978890:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251549.978898:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.1.11.93@o2ib10 00000800:00000200:12.0:1547251549.978901:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:13.0:1547251549.978903:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251549.978903:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251549.978904:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:7.2:1547251549.978908:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251549.978913:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:13.0:1547251549.978918:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251549.978920:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:13.0:1547251549.978921:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251549.978922:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000400:00000200:13.0:1547251549.978926:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251549.978927:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251549.978931:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03b60 00000400:00000200:12.0:1547251549.978931:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03b50 00000400:00000200:13.0:1547251549.978935:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f68d [1] + 192 00000400:00000200:12.0:1547251549.978948:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f67d [1] + 192 00000100:00000200:13.0:1547251549.978953:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a5d00 x1622384012442464/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251556 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251549.978963:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251549.978965:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000100:00000200:12.0:1547251549.978965:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824750000 x1622384012442448/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251556 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251549.978967:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:13.0:1547251549.978969:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:13.0:1547251549.978970:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:13.0:1547251549.978972:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:13.0:1547251549.978975:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251549.978976:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251549.978977:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)++ 00000100:00000200:14.0:1547251549.978978:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a5d00 x1622384012442464/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251556 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251549.978979:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251549.978979:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (24)-- 00000800:00000200:12.0:1547251549.978980:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251549.978982:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:14.0:1547251549.978983:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a5d00 x1622384012442464/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251556 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251549.978983:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03b70 00000400:00000200:14.0:1547251549.978988:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60088 00010000:00080000:14.0:1547251549.978991:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a5d00 x1622384012442464/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251556 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251549.978991:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824750000 x1622384012442448/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251556 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251549.978997:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824750000 x1622384012442448/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251556 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251549.979001:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00000400:00000200:13.0:1547251549.979004:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f6ad [1] + 192 00010000:00080000:8.0:1547251549.979005:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824750000 x1622384012442448/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251556 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:13.0:1547251549.979008:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328dbc00 x1622384012442480/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251556 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251549.979018:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251549.979019:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:13.0:1547251549.979020:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:13.0:1547251549.979021:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000100:00000200:2.0:1547251549.979027:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328dbc00 x1622384012442480/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251556 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251549.979033:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328dbc00 x1622384012442480/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251556 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251549.979037:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00010000:00080000:2.0:1547251549.979040:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328dbc00 x1622384012442480/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251556 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251552.090680:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442496, portal 4 00000100:00000200:9.0:1547251552.090682:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442496, offset 0 00000400:00000200:9.0:1547251552.090686:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251552.090688:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251552.090691:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22980] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251552.090694:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22979] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22980] 00000400:00000200:9.0:1547251552.090696:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251552.090698:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251552.090699:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:13.0:1547251552.090700:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442512, portal 4 00000400:00000200:9.0:1547251552.090700:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9725, 9726] 00000400:00000200:9.0:1547251552.090702:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000100:00000200:13.0:1547251552.090703:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442512, offset 0 00000400:00000200:13.0:1547251552.090706:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251552.090707:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000400:00000200:13.0:1547251552.090709:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:9.0:1547251552.090710:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:13.0:1547251552.090711:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22980] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:9.0:1547251552.090713:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251552.090714:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000400:00000200:13.0:1547251552.090715:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22980] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22980] 00000800:00000200:9.0:1547251552.090715:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000400:00000200:13.0:1547251552.090718:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:13.0:1547251552.090719:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251552.090719:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000400:00000200:13.0:1547251552.090721:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251552.090722:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9726, 9726] 00000400:00000200:13.0:1547251552.090724:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:13.0:1547251552.090729:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251552.090732:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:13.0:1547251552.090735:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251552.090736:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:13.0:1547251552.090737:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:13.0:1547251552.090740:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000100:00000200:0.0:1547251552.090749:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442528, portal 4 00000100:00000200:0.0:1547251552.090753:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442528, offset 0 00000400:00000200:0.0:1547251552.090759:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:0.0:1547251552.090762:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:0.0:1547251552.090764:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22981] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251552.090767:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22980] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22981] 00000800:00000200:7.2:1547251552.090769:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000400:00000200:0.0:1547251552.090770:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251552.090771:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251552.090773:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251552.090774:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9726, 9727] 00000400:00000200:0.0:1547251552.090776:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:0.0:1547251552.090780:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251552.090783:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:11.0:1547251552.090786:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:0.0:1547251552.090786:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251552.090787:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (22)++ 00000800:00000200:0.0:1547251552.090789:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (24)++ 00000800:00000200:0.0:1547251552.090791:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (25)-- 00000800:00000200:11.0:1547251552.090796:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000800:00000200:5.2:1547251552.090796:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000100:00000200:11.0:1547251552.090801:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf6000 x1622384012442496/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251552.090809:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaf68 00000400:00000200:11.0:1547251552.090810:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251552.090814:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251552.090814:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:10.0:1547251552.090816:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000800:00000200:11.0:1547251552.090817:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (24)-- 00000800:00000200:11.0:1547251552.090819:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251552.090820:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000100:00000200:10.0:1547251552.090821:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f7500 x1622384012442528/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251552.090823:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af5400 x1622384012442512/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251552.090827:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00000400:00000200:10.0:1547251552.090828:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251552.090830:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b990 00000400:00000200:11.0:1547251552.090831:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251552.090832:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251552.090834:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251552.090834:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251552.090835:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:11.0:1547251552.090836:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251552.090838:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:7.2:1547251552.090860:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:11.0:1547251552.090872:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251552.090878:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251552.090881:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251552.090882:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251552.090883:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251552.090884:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:7.2:1547251552.090893:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:5.2:1547251552.090898:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:10.0:1547251552.090905:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251552.090907:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251552.090908:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251552.090909:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251552.090912:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251552.090914:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251552.090917:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03b80 00000400:00000200:11.0:1547251552.090918:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03b90 00000400:00000200:10.0:1547251552.090921:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6549 [1] + 192 00000400:00000200:11.0:1547251552.090925:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6569 [1] + 192 00000100:00000200:10.0:1547251552.090929:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf6000 x1622384012442496/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251552.090947:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251552.090948:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af5400 x1622384012442512/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251552.090949:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251552.090950:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251552.090952:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251552.090953:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251552.090955:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251552.090959:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:10.0:1547251552.090960:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251552.090961:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000100:00000200:9.0:1547251552.090961:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf6000 x1622384012442496/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251552.090962:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000400:00000200:10.0:1547251552.090963:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03ba0 00000800:00000200:11.0:1547251552.090964:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251552.090966:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:9.0:1547251552.090966:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf6000 x1622384012442496/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251552.090968:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:9.0:1547251552.090970:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaa18 00000400:00000200:10.0:1547251552.090974:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6579 [1] + 192 00010000:00080000:9.0:1547251552.090974:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf6000 x1622384012442496/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:13.0:1547251552.090979:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af5400 x1622384012442512/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:13.0:1547251552.090985:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af5400 x1622384012442512/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251552.090989:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792bd48 00000100:00000200:10.0:1547251552.090992:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f7500 x1622384012442528/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:13.0:1547251552.090993:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af5400 x1622384012442512/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251552.091002:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251552.091003:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251552.091004:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251552.091005:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:0.0:1547251552.091011:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f7500 x1622384012442528/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251552.091017:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f7500 x1622384012442528/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251552.091021:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00010000:00080000:0.0:1547251552.091025:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f7500 x1622384012442528/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251552.554672:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442544, portal 4 00000100:00000200:8.0:1547251552.554676:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442544, offset 0 00000400:00000200:8.0:1547251552.554679:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251552.554682:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251552.554684:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22981] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251552.554687:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22981] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22981] 00000400:00000200:8.0:1547251552.554689:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251552.554691:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251552.554692:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251552.554693:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9719, 9719] 00000400:00000200:8.0:1547251552.554695:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:8.0:1547251552.554700:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251552.554702:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:8.0:1547251552.554705:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000100:00000200:0.0:1547251552.554705:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442576, portal 4 00000800:00000200:8.0:1547251552.554707:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:8.0:1547251552.554708:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000100:00000200:0.0:1547251552.554708:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442576, offset 0 00000800:00000200:8.0:1547251552.554711:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000400:00000200:0.0:1547251552.554711:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:0.0:1547251552.554714:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:0.0:1547251552.554716:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22982] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251552.554719:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22981] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22982] 00000400:00000200:0.0:1547251552.554722:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251552.554723:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:12.2:1547251552.554725:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000400:00000200:0.0:1547251552.554726:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251552.554727:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9719, 9720] 00000100:00000200:15.0:1547251552.554729:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442560, portal 4 00000400:00000200:0.0:1547251552.554729:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000100:00000200:15.0:1547251552.554732:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442560, offset 0 00000400:00000200:0.0:1547251552.554734:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:15.0:1547251552.554736:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:0.0:1547251552.554737:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:15.0:1547251552.554739:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:0.0:1547251552.554739:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251552.554741:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000400:00000200:15.0:1547251552.554742:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22982] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:0.0:1547251552.554742:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:15.0:1547251552.554745:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22982] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22982] 00000800:00000200:0.0:1547251552.554745:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:15.0:1547251552.554747:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:15.0:1547251552.554749:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251552.554751:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251552.554752:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9720, 9720] 00000400:00000200:15.0:1547251552.554754:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:0.2:1547251552.554756:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:15.0:1547251552.554759:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251552.554762:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:15.0:1547251552.554765:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251552.554767:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (22)++ 00000800:00000200:15.0:1547251552.554768:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00000800:00000200:15.0:1547251552.554771:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (24)-- 00000800:00000200:7.0:1547251552.554779:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:7.0:1547251552.554782:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000100:00000200:7.0:1547251552.554786:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824755d00 x1622384012442544/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251552.554791:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000400:00000200:7.0:1547251552.554792:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251552.554796:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:7.0:1547251552.554798:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:7.0:1547251552.554800:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:7.0:1547251552.554801:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:7.0:1547251552.554804:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328dbf00 x1622384012442576/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251552.554808:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665c38 00000400:00000200:7.0:1547251552.554809:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251552.554810:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000400:00000200:7.0:1547251552.554811:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:7.0:1547251552.554812:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251552.554813:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:7.0:1547251552.554814:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:7.0:1547251552.554815:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:4.0:1547251552.554816:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af3c00 x1622384012442560/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251552.554822:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1c38 00000400:00000200:4.0:1547251552.554823:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251552.554826:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251552.554828:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:12.2:1547251552.554882:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:0.2:1547251552.554887:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:0.2:1547251552.554892:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:7.0:1547251552.554894:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:7.0:1547251552.554897:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251552.554900:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251552.554900:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:4.0:1547251552.554903:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000400:00000200:7.0:1547251552.554904:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03bb0 00000400:00000200:4.0:1547251552.554906:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:7.0:1547251552.554907:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3275 [1] + 192 00000400:00000200:4.0:1547251552.554910:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03bd0 00000100:00000200:7.0:1547251552.554911:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824755d00 x1622384012442544/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251552.554922:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251552.554923:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3285 [1] + 192 00000800:00000200:7.0:1547251552.554924:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000800:00000200:7.0:1547251552.554925:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:7.0:1547251552.554926:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:4.0:1547251552.554927:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328dbf00 x1622384012442576/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:7.0:1547251552.554928:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:7.0:1547251552.554930:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251552.554933:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:7.0:1547251552.554935:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03bc0 00000400:00000200:4.0:1547251552.554944:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251552.554946:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824755d00 x1622384012442544/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251552.554946:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251552.554949:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251552.554950:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251552.554951:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:8.0:1547251552.554952:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824755d00 x1622384012442544/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251552.554953:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:4.0:1547251552.554954:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:8.0:1547251552.554957:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00000400:00000200:7.0:1547251552.554959:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3295 [1] + 192 00010000:00080000:8.0:1547251552.554960:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824755d00 x1622384012442544/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:0.0:1547251552.554963:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328dbf00 x1622384012442576/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251552.554968:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328dbf00 x1622384012442576/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251552.554972:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00000100:00000200:7.0:1547251552.554975:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af3c00 x1622384012442560/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251552.554976:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328dbf00 x1622384012442576/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:7.0:1547251552.554982:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:7.0:1547251552.554984:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:7.0:1547251552.554986:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:7.0:1547251552.554987:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000100:00000200:15.0:1547251552.554993:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af3c00 x1622384012442560/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:15.0:1547251552.554998:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af3c00 x1622384012442560/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251552.555003:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00010000:00080000:15.0:1547251552.555006:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af3c00 x1622384012442560/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251552.954707:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442592, portal 4 00000100:00000200:7.0:1547251552.954710:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442592, offset 0 00000400:00000200:7.0:1547251552.954712:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251552.954715:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251552.954718:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03be0 00000400:00000200:7.0:1547251552.954720:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1a25 [1] + 0 00000400:00000200:7.0:1547251552.954723:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990e58 00000100:00000200:7.0:1547251552.954724:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:8.0:1547251552.954731:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442608, portal 4 00000400:00000200:7.0:1547251552.954733:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990e58 00000100:00000200:8.0:1547251552.954734:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442608, offset 0 00000400:00000200:7.0:1547251552.954734:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251552.954736:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:7.0:1547251552.954737:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df6900 x1622384012442592/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251552.954739:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251552.954742:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03bf0 00000400:00000200:7.0:1547251552.954742:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:7.0:1547251552.954742:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251552.954743:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:8.0:1547251552.954744:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd3f79 [1] + 0 00000400:00000200:8.0:1547251552.954747:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ae6a914880 00000100:00000200:8.0:1547251552.954749:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251552.954758:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914880 00000400:00000200:8.0:1547251552.954759:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251552.954761:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960eec00 x1622384012442608/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251552.954764:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442592 00000400:00000200:8.0:1547251552.954766:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688770 00000400:00000200:8.0:1547251552.954767:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251552.954768:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00010000:00000200:5.0:1547251552.954773:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11a450 x1622384012442592/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:613/0 lens 224/368 e 0 to 0 dl 1547251558 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:10.0:1547251552.954779:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442608 00000100:00000200:5.0:1547251552.954780:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442592, offset 192 00000400:00000200:5.0:1547251552.954782:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251552.954786:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251552.954788:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03be0 00000400:00000200:5.0:1547251552.954791:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae8e1 [1] + 192 00010000:00000200:10.0:1547251552.954792:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3de450 x1622384012442608/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:613/0 lens 224/368 e 0 to 0 dl 1547251558 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251552.954794:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df6900 x1622384012442592/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:10.0:1547251552.954801:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442608, offset 192 00000400:00000200:10.0:1547251552.954804:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251552.954804:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251552.954806:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad110 00000400:00000200:10.0:1547251552.954807:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251552.954807:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251552.954808:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:10.0:1547251552.954810:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03bf0 00000100:00000200:7.0:1547251552.954812:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df6900 x1622384012442592/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251552.954813:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251552.954817:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df6900 x1622384012442592/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251552.954821:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00000400:00000200:10.0:1547251552.954823:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae8f1 [1] + 192 00010000:00080000:7.0:1547251552.954824:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df6900 x1622384012442592/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251552.954826:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960eec00 x1622384012442608/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251552.954835:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:10.0:1547251552.954837:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914550 00000400:00000200:10.0:1547251552.954837:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251552.954839:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251552.954843:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960eec00 x1622384012442608/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:10.0:1547251552.954844:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:8.0:1547251552.954848:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960eec00 x1622384012442608/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251552.954852:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00010000:00080000:8.0:1547251552.954855:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba960eec00 x1622384012442608/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251552.970688:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442624, portal 4 00000100:00000200:6.0:1547251552.970692:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442624, offset 0 00000400:00000200:6.0:1547251552.970695:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251552.970697:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251552.970700:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03c00 00000400:00000200:6.0:1547251552.970702:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1a55 [1] + 0 00000400:00000200:6.0:1547251552.970706:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b6459902a8 00000100:00000200:6.0:1547251552.970707:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251552.970716:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459902a8 00000400:00000200:6.0:1547251552.970717:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251552.970719:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37937b00 x1622384012442624/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251552.970724:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990bb0 00000400:00000200:6.0:1547251552.970724:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251552.970726:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251552.970748:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442624 00010000:00000200:4.0:1547251552.970757:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11a050 x1622384012442624/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:613/0 lens 224/368 e 0 to 0 dl 1547251558 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251552.970764:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442624, offset 192 00000400:00000200:4.0:1547251552.970766:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251552.970769:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251552.970772:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03c00 00000400:00000200:4.0:1547251552.970775:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae901 [1] + 192 00000100:00000200:4.0:1547251552.970778:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37937b00 x1622384012442624/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251552.970787:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251552.970789:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b2a8 00000400:00000200:4.0:1547251552.970790:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251552.970791:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251552.970794:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37937b00 x1622384012442624/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251552.970796:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251552.970798:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37937b00 x1622384012442624/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251552.970802:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00010000:00080000:6.0:1547251552.970805:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37937b00 x1622384012442624/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251559 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:0.2:1547251554.653389:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251554.653405:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:4.0:1547251554.653444:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:4.0:1547251554.653448:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251554.653452:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251554.653456:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b10230 00000400:00000200:4.0:1547251554.653459:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 56672 00000100:00000200:4.0:1547251554.653462:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:7.0:1547251554.653467:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:7.0:1547251554.653471:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251554.653472:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251554.653474:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251554.653475:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251554.653477:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:7.0:1547251554.653478:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b10220 00000800:00000200:4.0:1547251554.653478:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251554.653481:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:7.0:1547251554.653482:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 56896 00000800:00000200:4.0:1547251554.653484:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000100:00000200:7.0:1547251554.653485:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:4.0:1547251554.653488:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251554.653492:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 26 MB=0x5c38cb6b10200 00000400:00000200:7.0:1547251554.653496:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251554.653497:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0xd4a19 [1] + 0 00000800:00000200:7.0:1547251554.653498:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000400:00000200:4.0:1547251554.653500:0:12661:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52db28 00000800:00000200:7.0:1547251554.653502:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000100:00000200:4.0:1547251554.653502:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000800:00000200:7.0:1547251554.653504:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:7.0:1547251554.653506:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:7.0:1547251554.653508:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000100:00000200:5.0:1547251554.653509:0:13324:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444784 00000400:00000200:4.0:1547251554.653510:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52db28 00000400:00000200:4.0:1547251554.653511:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251554.653513:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251554.653513:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251554.653515:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:7.0:1547251554.653516:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b10210 00010000:00000200:5.0:1547251554.653516:0:13324:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b824448c00 x1622384011444784/t0(0) o400->cache1-MDT0000-lwp-OST0008_UUID@10.0.11.92@o2ib10:615/0 lens 224/224 e 0 to 0 dl 1547251560 ref 1 fl Interpret:H/0/0 rc 0/0 00000800:00000200:4.0:1547251554.653516:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251554.653518:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000400:00000200:7.0:1547251554.653520:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 57120 00000800:00000200:4.0:1547251554.653520:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:7.0:1547251554.653523:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:7.0:1547251554.653529:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:7.0:1547251554.653530:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000100:00000200:5.0:1547251554.653531:0:13324:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444784, offset 192 00000800:00000200:7.0:1547251554.653532:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:7.0:1547251554.653533:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000400:00000200:5.0:1547251554.653534:0:13324:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000100:00000200:6.0:1547251554.653535:0:12981:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444768 00000100:00000200:7.0:1547251554.653539:0:12979:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444752 00000400:00000200:5.0:1547251554.653539:0:13324:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00010000:00000200:6.0:1547251554.653541:0:12981:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae37933600 x1622384011444768/t0(0) o400->cache1-MDT0000-lwp-OST0006_UUID@10.0.11.92@o2ib10:615/0 lens 224/224 e 0 to 0 dl 1547251560 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:5.0:1547251554.653542:0:13324:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00010000:00000200:7.0:1547251554.653543:0:12979:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae37932d00 x1622384011444752/t0(0) o400->cache1-MDT0000-lwp-OST0007_UUID@10.0.11.92@o2ib10:615/0 lens 224/224 e 0 to 0 dl 1547251560 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:5.0:1547251554.653544:0:13324:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:5.0:1547251554.653545:0:13324:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9720, 9721] 00000400:00000200:5.0:1547251554.653548:0:13324:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000100:00000200:6.0:1547251554.653550:0:12981:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444768, offset 192 00000100:00000200:7.0:1547251554.653551:0:12979:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444752, offset 192 00000400:00000200:5.0:1547251554.653553:0:13324:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.1.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:7.0:1547251554.653554:0:12979:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:5.0:1547251554.653556:0:13324:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:7.0:1547251554.653558:0:12979:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:6.0:1547251554.653559:0:12981:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:7.0:1547251554.653561:0:12979:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251554.653563:0:12979:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251554.653565:0:12979:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9721, 9721] 00000400:00000200:7.0:1547251554.653567:0:12979:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000100:00000200:8.0:1547251554.653569:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444736 00000400:00000200:7.0:1547251554.653574:0:12979:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00010000:00000200:8.0:1547251554.653575:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2dc850 x1622384011444736/t0(0) o400->04d65b15-cf9c-7ded-611d-ffe7fe3f3abf@10.0.11.92@o2ib10:615/0 lens 224/224 e 0 to 0 dl 1547251560 ref 1 fl Interpret:/0/0 rc 0/0 00000800:00000200:7.0:1547251554.653578:0:12979:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:5.0:1547251554.653578:0:13324:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:6.0:1547251554.653579:0:12981:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:5.0:1547251554.653580:0:13324:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000100:00000200:8.0:1547251554.653581:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384011444736, offset 192 00000800:00000200:5.0:1547251554.653581:0:13324:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:6.0:1547251554.653582:0:12981:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:0.2:1547251554.653596:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (22)++ 00000400:00000200:8.0:1547251554.653604:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000800:00000200:4.0:1547251554.653606:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)++ 00000400:00000200:8.0:1547251554.653608:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00000400:00000200:6.0:1547251554.653609:0:12981:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251554.653610:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251554.653610:0:12981:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9721, 9722] 00000400:00000200:8.0:1547251554.653612:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251554.653613:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9721, 9722] 00000400:00000200:6.0:1547251554.653613:0:12981:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:8.0:1547251554.653615:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:5.0:1547251554.653617:0:13324:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:4.0:1547251554.653617:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (24)-- 00000400:00000200:6.0:1547251554.653619:0:12981:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:4.0:1547251554.653619:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ada18 00000400:00000200:4.0:1547251554.653621:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:6.0:1547251554.653623:0:12981:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:7.0:1547251554.653624:0:12979:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:4.0:1547251554.653624:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:7.0:1547251554.653625:0:12979:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000400:00000200:8.0:1547251554.653626:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.1.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:7.0:1547251554.653627:0:12979:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:8.0:1547251554.653629:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:7.0:1547251554.653630:0:12979:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:6.0:1547251554.653630:0:12981:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:4.0:1547251554.653630:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251554.653631:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:6.0:1547251554.653632:0:12981:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:8.0:1547251554.653633:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:6.0:1547251554.653633:0:12981:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:8.0:1547251554.653634:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:8.0:1547251554.653635:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:6.0:1547251554.653637:0:12981:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:8.0:1547251554.653638:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:12.2:1547251554.653641:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000100:00000200:8.0:1547251554.653642:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:0.2:1547251554.653646:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:12.2:1547251554.653651:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251554.653655:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251554.653656:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251554.653657:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251554.653658:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000400:00000200:4.0:1547251554.653659:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00000400:00000200:5.0:1547251554.653660:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00000400:00000200:4.0:1547251554.653660:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251554.653661:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251554.653664:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:5.0:1547251554.653665:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251554.653666:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251554.653667:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251554.653668:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251554.653669:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:4.0:1547251554.653669:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251554.653671:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000400:00000200:4.0:1547251554.653671:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00000800:00000200:5.0:1547251554.653672:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:4.0:1547251554.653672:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251554.653675:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251554.653676:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:7.2:1547251554.687704:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251554.687718:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:13.0:1547251554.687761:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251554.687765:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:13.0:1547251554.687766:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000800:00000200:12.0:1547251554.687768:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:13.0:1547251554.687771:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251554.687773:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251554.687777:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510210 00000400:00000200:12.0:1547251554.687777:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510220 00000400:00000200:13.0:1547251554.687781:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 27680 00000400:00000200:12.0:1547251554.687785:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 27904 00000100:00000200:13.0:1547251554.687789:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:13.0:1547251554.687799:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:12.0:1547251554.687799:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:13.0:1547251554.687802:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:13.0:1547251554.687804:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000400:00000200:12.0:1547251554.687804:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251554.687805:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251554.687806:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000800:00000200:13.0:1547251554.687807:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251554.687808:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:13.0:1547251554.687809:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000800:00000200:12.0:1547251554.687809:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251554.687810:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251554.687813:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:13.0:1547251554.687814:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251554.687817:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251554.687819:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 26 MB=0x5c38cb3510200 00000400:00000200:12.0:1547251554.687820:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510230 00000400:00000200:13.0:1547251554.687822:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0xd4a29 [1] + 0 00000400:00000200:12.0:1547251554.687823:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 28128 00000400:00000200:13.0:1547251554.687825:0:12666:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d088 00000100:00000200:12.0:1547251554.687826:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:13.0:1547251554.687827:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:12.0:1547251554.687829:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251554.687831:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000800:00000200:12.0:1547251554.687833:0:12665:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (23)++ 00000400:00000200:13.0:1547251554.687834:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d088 00000400:00000200:13.0:1547251554.687835:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251554.687836:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251554.687836:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (24)-- 00000800:00000200:12.0:1547251554.687837:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (23)-- 00000800:00000200:13.0:1547251554.687838:0:12666:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (23)++ 00000800:00000200:12.0:1547251554.687839:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:13.0:1547251554.687840:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251554.687840:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:13.0:1547251554.687841:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251554.687842:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000800:00000200:12.0:1547251554.687843:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:13.0:1547251554.687844:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:9.2:1547251554.687854:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000100:00000200:14.0:1547251554.687861:0:13374:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821648 00000100:00000200:13.0:1547251554.687864:0:13371:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821664 00000100:00000200:12.0:1547251554.687864:0:12987:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821680 00000100:00000200:8.0:1547251554.687867:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821632 00010000:00000200:14.0:1547251554.687870:0:13374:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b17b2dd100 x1622383954821648/t0(0) o400->cache1-MDT0000-lwp-OST000a_UUID@10.0.11.93@o2ib10:615/0 lens 224/224 e 0 to 0 dl 1547251560 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:13.0:1547251554.687870:0:13371:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97bae1af1500 x1622383954821664/t0(0) o400->cache1-MDT0000-lwp-OST0009_UUID@10.0.11.93@o2ib10:615/0 lens 224/224 e 0 to 0 dl 1547251560 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:12.0:1547251554.687870:0:12987:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97bae1af5100 x1622383954821680/t0(0) o400->cache1-MDT0000-lwp-OST000b_UUID@10.0.11.93@o2ib10:615/0 lens 224/224 e 0 to 0 dl 1547251560 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:8.0:1547251554.687875:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2d9c50 x1622383954821632/t0(0) o400->b99b3dbb-ff3e-ba67-561d-3f0ab43da035@10.0.11.93@o2ib10:615/0 lens 224/224 e 0 to 0 dl 1547251560 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:13.0:1547251554.687879:0:13371:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821664, offset 192 00000100:00000200:12.0:1547251554.687879:0:12987:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821680, offset 192 00000100:00000200:14.0:1547251554.687880:0:13374:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821648, offset 192 00000400:00000200:12.0:1547251554.687882:0:12987:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000100:00000200:8.0:1547251554.687882:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954821632, offset 192 00000400:00000200:13.0:1547251554.687885:0:13371:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:8.0:1547251554.687885:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000400:00000200:12.0:1547251554.687886:0:12987:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:12.0:1547251554.687889:0:12987:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251554.687889:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00000400:00000200:14.0:1547251554.687890:0:13374:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000400:00000200:12.0:1547251554.687891:0:12987:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251554.687891:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251554.687893:0:12987:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9721, 9721] 00000400:00000200:8.0:1547251554.687893:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251554.687894:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9721, 9721] 00000400:00000200:12.0:1547251554.687896:0:12987:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:8.0:1547251554.687896:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251554.687902:0:12987:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251554.687906:0:12987:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:13.0:1547251554.687907:0:13371:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:13.0:1547251554.687910:0:13371:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251554.687911:0:13371:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251554.687913:0:13371:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9721, 9723] 00000400:00000200:13.0:1547251554.687915:0:13371:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:13.0:1547251554.687922:0:13371:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251554.687925:0:13371:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:14.0:1547251554.687927:0:13374:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00000400:00000200:14.0:1547251554.687929:0:13374:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251554.687931:0:13374:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251554.687932:0:13374:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9722, 9723] 00000400:00000200:14.0:1547251554.687934:0:13374:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251554.687947:0:13374:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.1.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251554.687950:0:13374:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:8.0:1547251554.687954:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.1.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251554.687957:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:14.0:1547251554.687958:0:13374:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251554.687958:0:13371:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251554.687958:0:12987:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251554.687960:0:13374:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:13.0:1547251554.687960:0:13371:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:12.0:1547251554.687960:0:12987:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:8.0:1547251554.687960:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251554.687961:0:13374:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (23)++ 00000800:00000200:13.0:1547251554.687961:0:13371:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251554.687961:0:12987:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:8.0:1547251554.687961:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:8.0:1547251554.687962:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:14.0:1547251554.687964:0:13374:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (24)-- 00000800:00000200:13.0:1547251554.687964:0:13371:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251554.687964:0:12987:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:8.0:1547251554.687965:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:8.0:1547251554.687969:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:12.0:1547251554.687973:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (23)++ 00000800:00000200:12.0:1547251554.687975:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (24)-- 00000800:00000200:7.2:1547251554.687976:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251554.687978:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:7.2:1547251554.687979:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:13.0:1547251554.687980:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251554.687981:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:13.0:1547251554.687982:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251554.687983:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000800:00000200:9.2:1547251554.687983:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:13.0:1547251554.687984:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60088 00000400:00000200:13.0:1547251554.687985:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251554.687985:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038198 00000400:00000200:12.0:1547251554.687986:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251554.687988:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:12.0:1547251554.687990:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251554.687992:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251554.687993:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:13.0:1547251554.687994:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:13.0:1547251554.687995:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251554.687995:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000400:00000200:13.0:1547251554.687996:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792bd48 00000800:00000200:12.0:1547251554.687996:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000400:00000200:13.0:1547251554.687997:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:12.0:1547251554.687998:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000400:00000200:13.0:1547251554.688000:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251554.688000:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:13.0:1547251554.688002:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:12.0:1547251554.688002:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688e58 00000800:00000200:13.0:1547251554.688003:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000400:00000200:12.0:1547251554.688003:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:13.0:1547251554.688005:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:12.0:1547251554.688006:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251554.688007:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:5.2:1547251554.719422:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251554.719437:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:11.0:1547251554.719477:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251554.719481:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251554.719482:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251554.719485:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251554.719487:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251554.719489:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251554.719492:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510260 00000400:00000200:10.0:1547251554.719494:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510250 00000400:00000200:11.0:1547251554.719496:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 11424 00000400:00000200:10.0:1547251554.719500:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 11648 00000100:00000200:11.0:1547251554.719504:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:11.0:1547251554.719514:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251554.719514:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:11.0:1547251554.719517:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251554.719519:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000400:00000200:10.0:1547251554.719519:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251554.719521:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251554.719521:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251554.719522:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251554.719523:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251554.719524:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:11.0:1547251554.719525:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251554.719525:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251554.719528:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251554.719529:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251554.719532:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 26 MB=0x5c38cb3510230 00000400:00000200:10.0:1547251554.719532:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251554.719535:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510240 00000400:00000200:11.0:1547251554.719536:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0xd4a69 [1] + 0 00000400:00000200:11.0:1547251554.719539:0:12663:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52dee0 00000400:00000200:10.0:1547251554.719541:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 11872 00000100:00000200:11.0:1547251554.719544:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:11.0:1547251554.719550:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52dee0 00000400:00000200:11.0:1547251554.719551:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251554.719551:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:11.0:1547251554.719553:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251554.719554:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251554.719555:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251554.719557:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000400:00000200:10.0:1547251554.719558:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251554.719559:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251554.719559:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251554.719563:0:12664:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251554.719565:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251554.719566:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000100:00000200:11.0:1547251554.719576:0:12983:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821728 00000800:00000200:7.2:1547251554.719577:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000100:00000200:9.0:1547251554.719578:0:12982:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821712 00000100:00000200:8.0:1547251554.719580:0:13370:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821696 00010000:00000200:11.0:1547251554.719582:0:12983:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ba960e9200 x1622383954821728/t0(0) o400->cache1-MDT0000-lwp-OST0004_UUID@10.0.11.91@o2ib10:615/0 lens 224/224 e 0 to 0 dl 1547251560 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:9.0:1547251554.719599:0:12982:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b17b07c500 x1622383954821712/t0(0) o400->cache1-MDT0000-lwp-OST0003_UUID@10.0.11.91@o2ib10:615/0 lens 224/224 e 0 to 0 dl 1547251560 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:8.0:1547251554.719599:0:13370:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b17b07c200 x1622383954821696/t0(0) o400->cache1-MDT0000-lwp-OST0005_UUID@10.0.11.91@o2ib10:615/0 lens 224/224 e 0 to 0 dl 1547251560 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:9.0:1547251554.719607:0:12982:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821712, offset 192 00000100:00000200:8.0:1547251554.719608:0:13370:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821696, offset 192 00000100:00000200:11.0:1547251554.719610:0:12983:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821728, offset 192 00000400:00000200:9.0:1547251554.719611:0:12982:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000100:00000200:12.0:1547251554.719614:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821680 00000400:00000200:8.0:1547251554.719615:0:13370:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000400:00000200:9.0:1547251554.719616:0:12982:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000400:00000200:11.0:1547251554.719618:0:12983:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251554.719619:0:12982:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00010000:00000200:12.0:1547251554.719620:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2da850 x1622383954821680/t0(0) o400->dc7951f7-3e71-e9c4-2e5e-5f95f4837d68@10.0.11.91@o2ib10:615/0 lens 224/224 e 0 to 0 dl 1547251560 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:9.0:1547251554.719621:0:12982:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251554.719623:0:12982:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9727, 9727] 00000400:00000200:9.0:1547251554.719625:0:12982:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000100:00000200:12.0:1547251554.719628:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954821680, offset 192 00000400:00000200:12.0:1547251554.719631:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251554.719632:0:12982:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.1.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000400:00000200:12.0:1547251554.719634:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:12.0:1547251554.719636:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251554.719636:0:12982:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251554.719637:0:13370:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000400:00000200:12.0:1547251554.719638:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251554.719639:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9727, 9728] 00000400:00000200:8.0:1547251554.719640:0:13370:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251554.719641:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:8.0:1547251554.719642:0:13370:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251554.719644:0:13370:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9728, 9728] 00000400:00000200:8.0:1547251554.719646:0:13370:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:8.0:1547251554.719652:0:13370:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.91@o2ib10(10.1.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251554.719655:0:13370:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:11.0:1547251554.719656:0:12983:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:11.0:1547251554.719658:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251554.719661:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251554.719662:0:12983:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9729, 9728] 00000400:00000200:11.0:1547251554.719664:0:12983:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251554.719669:0:12983:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251554.719672:0:12983:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:9.0:1547251554.719673:0:12982:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251554.719675:0:12982:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000400:00000200:12.0:1547251554.719676:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251554.719676:0:12982:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:12.0:1547251554.719679:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:9.0:1547251554.719679:0:12982:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251554.719680:0:12983:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251554.719681:0:12983:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:8.0:1547251554.719681:0:13370:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251554.719682:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251554.719682:0:12983:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:8.0:1547251554.719682:0:13370:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (22)++ 00000800:00000200:8.0:1547251554.719683:0:13370:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (23)++ 00000800:00000200:12.0:1547251554.719684:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:11.0:1547251554.719684:0:12983:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000800:00000200:12.0:1547251554.719685:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000800:00000200:8.0:1547251554.719687:0:13370:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (24)-- 00000800:00000200:12.0:1547251554.719688:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:7.2:1547251554.719690:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000100:00000200:12.0:1547251554.719692:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:5.2:1547251554.719696:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251554.719698:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:5.2:1547251554.719700:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251554.719701:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000800:00000200:11.0:1547251554.719703:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251554.719704:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:11.0:1547251554.719706:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000400:00000200:11.0:1547251554.719708:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaa18 00000800:00000200:10.0:1547251554.719708:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000400:00000200:11.0:1547251554.719709:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251554.719710:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000400:00000200:10.0:1547251554.719711:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bb28 00000400:00000200:11.0:1547251554.719712:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251554.719713:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251554.719715:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251554.719716:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000400:00000200:10.0:1547251554.719716:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251554.719717:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251554.719718:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:11.0:1547251554.719719:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688088 00000400:00000200:11.0:1547251554.719720:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251554.719720:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000800:00000200:10.0:1547251554.719721:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000400:00000200:11.0:1547251554.719722:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251554.719723:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038198 00000800:00000200:11.0:1547251554.719724:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:10.0:1547251554.719725:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251554.719726:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:11.0:1547251554.719728:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000400:00000200:10.0:1547251554.719728:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251554.719729:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251554.719730:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:10.0:1547251554.719731:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:15.0:1547251554.986677:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442640, portal 4 00000100:00000200:15.0:1547251554.986680:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442640, offset 0 00000400:00000200:15.0:1547251554.986683:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:15.0:1547251554.986686:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:15.0:1547251554.986688:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22983] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251554.986691:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22982] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22983] 00000400:00000200:15.0:1547251554.986694:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:15.0:1547251554.986695:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251554.986697:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:3.0:1547251554.986697:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442672, portal 4 00000400:00000200:15.0:1547251554.986698:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9723, 9723] 00000100:00000200:9.0:1547251554.986698:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442656, portal 4 00000400:00000200:15.0:1547251554.986700:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000100:00000200:3.0:1547251554.986700:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442672, offset 0 00000100:00000200:9.0:1547251554.986701:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442656, offset 0 00000400:00000200:3.0:1547251554.986703:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:15.0:1547251554.986704:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000400:00000200:9.0:1547251554.986704:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:3.0:1547251554.986706:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000800:00000200:15.0:1547251554.986707:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:9.0:1547251554.986707:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:9.0:1547251554.986709:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22983] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251554.986709:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22983] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:15.0:1547251554.986710:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251554.986712:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000400:00000200:3.0:1547251554.986712:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22983] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22983] 00000400:00000200:9.0:1547251554.986713:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22983] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22983] 00000800:00000200:15.0:1547251554.986714:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:9.0:1547251554.986715:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251554.986715:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251554.986716:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:15.0:1547251554.986717:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:9.0:1547251554.986717:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251554.986718:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251554.986719:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251554.986719:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9723, 9724] 00000400:00000200:9.0:1547251554.986720:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9723, 9724] 00000400:00000200:3.0:1547251554.986721:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000800:00000200:9.2:1547251554.986724:0:12680:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:3.0:1547251554.986726:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251554.986729:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:3.0:1547251554.986731:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251554.986733:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:3.0:1547251554.986734:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:9.0:1547251554.986735:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000800:00000200:3.0:1547251554.986736:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:9.0:1547251554.986739:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251554.986742:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:9.0:1547251554.986745:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251554.986747:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:13.0:1547251554.986748:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:9.0:1547251554.986748:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:9.0:1547251554.986751:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:13.0:1547251554.986757:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:13.0:1547251554.986763:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8ca8600 x1622384012442640/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251561 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251554.986765:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:13.0:1547251554.986770:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1220 00000400:00000200:13.0:1547251554.986772:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251554.986776:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251554.986779:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:7.2:1547251554.986786:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:7.2:1547251554.986796:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251554.986803:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251554.986806:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:13.0:1547251554.986808:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:13.0:1547251554.986810:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000100:00000200:12.0:1547251554.986810:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c5400 x1622384012442672/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251561 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:13.0:1547251554.986813:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf2400 x1622384012442656/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251561 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251554.986817:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555088 00000400:00000200:12.0:1547251554.986818:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251554.986819:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaf68 00000400:00000200:13.0:1547251554.986820:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251554.986821:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:13.0:1547251554.986823:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251554.986824:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:13.0:1547251554.986825:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251554.986826:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:13.0:1547251554.986827:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:7.2:1547251554.986883:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251554.986888:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.2:1547251554.986893:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:12.0:1547251554.986896:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251554.986900:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.93@o2ib10 00000800:00000200:13.0:1547251554.986901:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:13.0:1547251554.986903:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251554.986905:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251554.986908:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251554.986910:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03c10 00000400:00000200:13.0:1547251554.986911:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03c20 00000400:00000200:12.0:1547251554.986914:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f71d [1] + 192 00000400:00000200:13.0:1547251554.986919:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f74d [1] + 192 00000100:00000200:12.0:1547251554.986923:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8ca8600 x1622384012442640/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251561 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251554.986948:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:13.0:1547251554.986950:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf2400 x1622384012442656/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251561 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251554.986951:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000800:00000200:12.0:1547251554.986953:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:12.0:1547251554.986954:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251554.986956:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251554.986959:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251554.986963:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251554.986966:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:12.0:1547251554.986967:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03c30 00000800:00000200:13.0:1547251554.986968:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000100:00000200:15.0:1547251554.986969:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8ca8600 x1622384012442640/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251561 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251554.986970:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:13.0:1547251554.986971:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:13.0:1547251554.986972:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:15.0:1547251554.986974:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8ca8600 x1622384012442640/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251561 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251554.986974:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:13.0:1547251554.986975:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:15.0:1547251554.986978:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00000400:00000200:12.0:1547251554.986981:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f73d [1] + 192 00010000:00080000:15.0:1547251554.986982:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8ca8600 x1622384012442640/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251561 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251554.986987:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf2400 x1622384012442656/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251561 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251554.986992:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf2400 x1622384012442656/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251561 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251554.986996:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaa18 00000100:00000200:12.0:1547251554.986999:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c5400 x1622384012442672/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251561 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251554.987000:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf2400 x1622384012442656/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251561 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:12.0:1547251554.987008:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251554.987010:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251554.987011:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251554.987012:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000100:00000200:3.0:1547251554.987063:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c5400 x1622384012442672/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251561 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251554.987069:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c5400 x1622384012442672/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251561 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251554.987073:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251554.987077:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c5400 x1622384012442672/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251561 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251556.346710:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012442704, portal 10 00000100:00000200:4.0:1547251556.346714:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012442704, offset 0 00000400:00000200:4.0:1547251556.346717:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251556.346720:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251556.346723:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c03c50 00000400:00000200:4.0:1547251556.346725:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 57344 00000100:00000200:4.0:1547251556.346729:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:7.0:1547251556.346733:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012442720, portal 10 00000100:00000200:6.0:1547251556.346733:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012442688, portal 25 00000100:00000200:7.0:1547251556.346738:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012442720, offset 0 00000100:00000200:6.0:1547251556.346738:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 26, xid 1622384012442688, offset 0 00000400:00000200:4.0:1547251556.346738:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251556.346741:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c434b00 x1622384012442704/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251556.346748:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b880 00000400:00000200:4.0:1547251556.346750:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251556.346751:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251556.346751:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251556.346754:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251556.346756:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251556.346756:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:5.0:1547251556.346758:0:12981:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442704 00000400:00000200:7.0:1547251556.346760:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c03c60 00000400:00000200:6.0:1547251556.346760:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 26 MB=0x5c38cb6c03c40 00000400:00000200:7.0:1547251556.346764:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 57568 00000400:00000200:6.0:1547251556.346764:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-0@lo of length 224/224 into md 0xd4a79 [1] + 0 00010000:00000200:5.0:1547251556.346765:0:12981:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444b600 x1622384012442704/t0(0) o400->cache1-MDT0000-lwp-MDT0000_UUID@0@lo:617/0 lens 224/224 e 0 to 0 dl 1547251562 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:7.0:1547251556.346767:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:6.0:1547251556.346767:0:12675:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52de58 00000100:00000200:6.0:1547251556.346769:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:6.0:1547251556.346778:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52de58 00000400:00000200:6.0:1547251556.346779:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251556.346780:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251556.346781:0:12981:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012442704, offset 192 00000100:00000200:6.0:1547251556.346782:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c432100 x1622384012442688/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251563 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251556.346788:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459902a8 00000400:00000200:6.0:1547251556.346789:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:7.0:1547251556.346790:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c432700 x1622384012442720/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251556.346791:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251556.346796:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00000100:00000200:6.0:1547251556.346796:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012442752, portal 10 00000400:00000200:7.0:1547251556.346797:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:4.0:1547251556.346798:0:12979:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442720 00000400:00000200:7.0:1547251556.346799:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:5.0:1547251556.346799:0:12981:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:6.0:1547251556.346801:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012442752, offset 0 00010000:00000200:4.0:1547251556.346802:0:12979:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae37936f00 x1622384012442720/t0(0) o400->cache1-MDT0000-lwp-OST0000_UUID@0@lo:617/0 lens 224/224 e 0 to 0 dl 1547251562 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:6.0:1547251556.346803:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251556.346803:0:12981:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251556.346806:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251556.346807:0:12981:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c03c50 00000100:00000200:7.0:1547251556.346809:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012442736, portal 10 00000400:00000200:6.0:1547251556.346809:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c03c80 00000100:00000200:4.0:1547251556.346810:0:12979:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012442720, offset 192 00000400:00000200:5.0:1547251556.346811:0:12981:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xae921 [1] + 192 00000400:00000200:6.0:1547251556.346812:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 57792 00000100:00000200:5.0:1547251556.346815:0:12981:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c434b00 x1622384012442704/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251556.346817:0:12979:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:7.0:1547251556.346818:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012442736, offset 0 00000100:00000200:6.0:1547251556.346819:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:4.0:1547251556.346820:0:12979:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251556.346822:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251556.346823:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251556.346823:0:12979:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c03c60 00000400:00000200:7.0:1547251556.346825:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251556.346826:0:12981:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adc38 00000400:00000200:4.0:1547251556.346826:0:12979:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xae931 [1] + 192 00000400:00000200:7.0:1547251556.346827:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251556.346827:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:6.0:1547251556.346828:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c431500 x1622384012442752/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251556.346829:0:12981:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251556.346829:0:12979:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c432700 x1622384012442720/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251556.346830:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c03c70 00000400:00000200:6.0:1547251556.346833:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459904c8 00000400:00000200:6.0:1547251556.346834:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251556.346835:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 58016 00000400:00000200:6.0:1547251556.346836:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251556.346837:0:12979:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:14.0:1547251556.346838:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442688 00000100:00000200:7.0:1547251556.346838:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:7.0:1547251556.346843:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251556.346843:0:12979:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b880 00010000:00000200:14.0:1547251556.346844:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2dd850 x1622384012442688/t0(0) o400->a65be517-33e7-fa36-0de3-da9e02df4f8d@0@lo:617/0 lens 224/224 e 0 to 0 dl 1547251562 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:4.0:1547251556.346844:0:12979:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:7.0:1547251556.346845:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c430900 x1622384012442736/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251556.346845:0:12979:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251556.346847:0:12981:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442752 00000100:00000200:6.0:1547251556.346850:0:13355:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442736 00000100:00000200:14.0:1547251556.346851:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384012442688, offset 192 00000400:00000200:7.0:1547251556.346851:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea4c8 00010000:00000200:5.0:1547251556.346851:0:12981:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a765df1b00 x1622384012442752/t0(0) o400->cache1-MDT0000-lwp-OST0001_UUID@0@lo:617/0 lens 224/224 e 0 to 0 dl 1547251562 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:7.0:1547251556.346852:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251556.346853:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251556.346854:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c434b00 x1622384012442704/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251556.346855:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00010000:00000200:6.0:1547251556.346855:0:13355:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae37930300 x1622384012442736/t0(0) o400->cache1-MDT0000-lwp-OST0002_UUID@0@lo:617/0 lens 224/224 e 0 to 0 dl 1547251562 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:14.0:1547251556.346858:0:12929:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:4.0:1547251556.346860:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c434b00 x1622384012442704/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251556.346861:0:12929:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 25 MB=0x5c38cb6c03c40 00000100:00000200:6.0:1547251556.346862:0:13355:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012442736, offset 192 00000100:00000200:5.0:1547251556.346863:0:12981:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012442752, offset 192 00000400:00000200:6.0:1547251556.346866:0:13355:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251556.346866:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b990 00000100:00000200:7.0:1547251556.346868:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c432700 x1622384012442720/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251556.346868:0:13355:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251556.346868:0:12981:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251556.346870:0:12981:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251556.346871:0:13355:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c03c70 00000400:00000200:5.0:1547251556.346873:0:12981:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c03c80 00000100:00000200:7.0:1547251556.346874:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c432700 x1622384012442720/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251556.346879:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:14.0:1547251556.346881:0:12929:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 19 from 12345-0@lo of length 192/192 into md 0xae941 [1] + 192 00000400:00000200:6.0:1547251556.346884:0:13355:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xae961 [1] + 192 00000400:00000200:5.0:1547251556.346888:0:12981:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xae951 [1] + 192 00000100:00000200:14.0:1547251556.346891:0:12929:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c432100 x1622384012442688/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251556.346897:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:14.0:1547251556.346899:0:12929:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60088 00000100:00000200:6.0:1547251556.346899:0:13355:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c430900 x1622384012442736/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251556.346900:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251556.346901:0:12929:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:14.0:1547251556.346906:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000400:00000200:6.0:1547251556.346910:0:13355:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251556.346911:0:12981:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c431500 x1622384012442752/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251556.346912:0:13355:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459904c8 00000400:00000200:6.0:1547251556.346913:0:13355:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251556.346915:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251556.346916:0:13355:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:5.0:1547251556.346916:0:12981:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adc38 00000400:00000200:5.0:1547251556.346917:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251556.346918:0:12981:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251556.346920:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c430900 x1622384012442736/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251556.346925:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c430900 x1622384012442736/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251556.346930:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00000100:00000200:6.0:1547251556.346933:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c432100 x1622384012442688/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251556.346949:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c432100 x1622384012442688/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251556.346954:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990bb0 00000100:00000200:6.0:1547251556.346958:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c431500 x1622384012442752/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251556.346964:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c431500 x1622384012442752/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251563 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251556.346969:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459902a8 00000100:00000200:8.0:1547251557.098697:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442800, portal 4 00000100:00000200:8.0:1547251557.098700:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442800, offset 0 00000400:00000200:8.0:1547251557.098703:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251557.098706:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000100:00000200:2.0:1547251557.098707:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442768, portal 4 00000400:00000200:8.0:1547251557.098708:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22985] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:2.0:1547251557.098710:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442768, offset 0 00000400:00000200:8.0:1547251557.098711:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22983] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22985] 00000400:00000200:8.0:1547251557.098713:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:8.0:1547251557.098715:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251557.098715:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251557.098716:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251557.098718:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9729, 9729] 00000400:00000200:2.0:1547251557.098718:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251557.098719:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:2.0:1547251557.098720:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22985] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251557.098724:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000400:00000200:2.0:1547251557.098725:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22984] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22985] 00000800:00000200:8.0:1547251557.098727:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:2.0:1547251557.098727:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:2.0:1547251557.098729:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251557.098730:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000400:00000200:2.0:1547251557.098730:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251557.098731:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000400:00000200:2.0:1547251557.098731:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9729, 9730] 00000800:00000200:8.0:1547251557.098732:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:2.0:1547251557.098733:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000100:00000200:14.0:1547251557.098734:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442784, portal 4 00000800:00000200:8.0:1547251557.098735:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000100:00000200:14.0:1547251557.098737:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442784, offset 0 00000400:00000200:2.0:1547251557.098738:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251557.098741:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:14.0:1547251557.098742:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:2.0:1547251557.098744:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000400:00000200:14.0:1547251557.098745:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:2.0:1547251557.098745:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:2.0:1547251557.098746:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000400:00000200:14.0:1547251557.098747:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22986] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:2.0:1547251557.098749:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)-- 00000400:00000200:14.0:1547251557.098751:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22984] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22986] 00000400:00000200:14.0:1547251557.098753:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:14.0:1547251557.098754:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251557.098756:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251557.098757:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9730, 9730] 00000400:00000200:14.0:1547251557.098759:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:14.0:1547251557.098763:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251557.098766:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:14.0:1547251557.098769:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251557.098771:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:14.0:1547251557.098772:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:14.0:1547251557.098774:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:7.2:1547251557.098784:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:11.0:1547251557.098800:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:11.0:1547251557.098809:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (24)-- 00000800:00000200:5.2:1547251557.098809:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00000100:00000200:11.0:1547251557.098814:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824756900 x1622384012442800/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251557.098822:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688e58 00000400:00000200:11.0:1547251557.098823:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251557.098827:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251557.098827:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:10.0:1547251557.098829:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (24)-- 00000800:00000200:11.0:1547251557.098831:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:11.0:1547251557.098833:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000100:00000200:10.0:1547251557.098833:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f0000 x1622384012442784/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251557.098835:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000100:00000200:11.0:1547251557.098838:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b5100 x1622384012442768/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251557.098839:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:10.0:1547251557.098840:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251557.098843:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a5d8 00000400:00000200:11.0:1547251557.098844:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251557.098844:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251557.098846:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:11.0:1547251557.098847:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251557.098848:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:11.0:1547251557.098849:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251557.098850:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:5.2:1547251557.098909:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251557.098915:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:11.0:1547251557.098922:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251557.098926:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251557.098928:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000400:00000200:11.0:1547251557.098931:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251557.098931:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251557.098936:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03cb0 00000400:00000200:10.0:1547251557.098936:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251557.098949:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6669 [1] + 192 00000400:00000200:10.0:1547251557.098949:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03c90 00000400:00000200:10.0:1547251557.098953:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6689 [1] + 192 00000100:00000200:11.0:1547251557.098958:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824756900 x1622384012442800/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251557.098969:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251557.098971:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000100:00000200:10.0:1547251557.098971:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b5100 x1622384012442768/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251557.098973:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251557.098974:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251557.098976:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251557.098978:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251557.098982:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251557.098982:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251557.098984:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000100:00000200:8.0:1547251557.098984:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824756900 x1622384012442800/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251557.098985:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03ca0 00000800:00000200:10.0:1547251557.098986:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251557.098987:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251557.098989:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:8.0:1547251557.098989:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824756900 x1622384012442800/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251557.098990:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:8.0:1547251557.098993:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688088 00000400:00000200:11.0:1547251557.098996:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6699 [1] + 192 00010000:00080000:8.0:1547251557.098997:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824756900 x1622384012442800/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:2.0:1547251557.099002:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b5100 x1622384012442768/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251557.099007:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b5100 x1622384012442768/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251557.099011:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00000100:00000200:11.0:1547251557.099014:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f0000 x1622384012442784/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:2.0:1547251557.099015:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b5100 x1622384012442768/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:11.0:1547251557.099022:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251557.099023:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251557.099024:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251557.099025:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:14.0:1547251557.099032:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f0000 x1622384012442784/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:14.0:1547251557.099037:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f0000 x1622384012442784/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251557.099043:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00010000:00080000:14.0:1547251557.099046:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f0000 x1622384012442784/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:14.0:1547251557.562699:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442832, portal 4 00000100:00000200:14.0:1547251557.562701:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442832, offset 0 00000100:00000200:9.0:1547251557.562701:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442848, portal 4 00000100:00000200:9.0:1547251557.562703:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442848, offset 0 00000400:00000200:14.0:1547251557.562705:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251557.562707:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:14.0:1547251557.562708:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251557.562709:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:14.0:1547251557.562710:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22986] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251557.562712:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22986] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:14.0:1547251557.562713:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22985] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22986] 00000400:00000200:9.0:1547251557.562715:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22985] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22986] 00000400:00000200:14.0:1547251557.562716:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:14.0:1547251557.562717:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251557.562718:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000100:00000200:2.0:1547251557.562718:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442816, portal 4 00000400:00000200:14.0:1547251557.562719:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251557.562719:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251557.562720:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9722, 9723] 00000400:00000200:9.0:1547251557.562721:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251557.562722:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:9.0:1547251557.562722:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9722, 9723] 00000100:00000200:2.0:1547251557.562722:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442816, offset 0 00000400:00000200:9.0:1547251557.562724:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:2.0:1547251557.562726:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:14.0:1547251557.562727:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251557.562729:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:2.0:1547251557.562730:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:14.0:1547251557.562732:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:2.0:1547251557.562732:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22987] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251557.562733:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251557.562734:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:14.0:1547251557.562735:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:2.0:1547251557.562735:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22986] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22987] 00000800:00000200:9.0:1547251557.562736:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:14.0:1547251557.562738:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:2.0:1547251557.562738:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:9.0:1547251557.562739:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:2.0:1547251557.562739:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251557.562741:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251557.562742:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000400:00000200:2.0:1547251557.562742:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9723, 9724] 00000800:00000200:9.0:1547251557.562743:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000400:00000200:2.0:1547251557.562744:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:9.0:1547251557.562746:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000400:00000200:2.0:1547251557.562750:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251557.562752:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:0.2:1547251557.562752:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:2.0:1547251557.562756:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251557.562757:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:2.0:1547251557.562758:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (23)++ 00000800:00000200:12.2:1547251557.562761:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:2.0:1547251557.562761:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:4.0:1547251557.562809:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)++ 00000800:00000200:5.0:1547251557.562813:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251557.562813:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:5.0:1547251557.562815:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000100:00000200:4.0:1547251557.562817:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a3600 x1622384012442832/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251557.562819:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf2700 x1622384012442848/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251557.562825:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea660 00000400:00000200:4.0:1547251557.562825:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf606e8 00000400:00000200:5.0:1547251557.562827:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251557.562827:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251557.562831:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251557.562831:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251557.562833:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251557.562835:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251557.562835:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251557.562836:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:5.0:1547251557.562837:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251557.562838:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:5.0:1547251557.562839:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b5400 x1622384012442816/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251557.562845:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a6e8 00000400:00000200:5.0:1547251557.562846:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251557.562849:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251557.562851:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:0.2:1547251557.562918:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251557.562924:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:4.0:1547251557.562932:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:12.2:1547251557.562935:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:4.0:1547251557.562936:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.1.11.92@o2ib10 00000800:00000200:5.0:1547251557.562947:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251557.562950:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251557.562952:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251557.562955:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251557.562957:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03cd0 00000400:00000200:5.0:1547251557.562958:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03ce0 00000400:00000200:4.0:1547251557.562961:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe33c5 [1] + 192 00000400:00000200:5.0:1547251557.562965:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe33d5 [1] + 192 00000100:00000200:4.0:1547251557.562970:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a3600 x1622384012442832/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251557.562983:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251557.562985:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf2700 x1622384012442848/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251557.562985:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251557.562987:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251557.562988:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251557.562990:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:4.0:1547251557.562992:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251557.562994:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251557.562996:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251557.562997:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000100:00000200:14.0:1547251557.562998:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a3600 x1622384012442832/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251557.562999:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000400:00000200:4.0:1547251557.562999:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:5.0:1547251557.563000:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251557.563002:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000400:00000200:4.0:1547251557.563002:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03cc0 00000100:00000200:14.0:1547251557.563004:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a3600 x1622384012442832/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251557.563004:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:14.0:1547251557.563008:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00000100:00000200:9.0:1547251557.563011:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf2700 x1622384012442848/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:14.0:1547251557.563012:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a3600 x1622384012442832/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251557.563017:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf2700 x1622384012442848/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251557.563021:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaa18 00000400:00000200:4.0:1547251557.563024:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe33e5 [1] + 192 00010000:00080000:9.0:1547251557.563025:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf2700 x1622384012442848/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251557.563028:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b5400 x1622384012442816/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251557.563036:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251557.563038:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251557.563039:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251557.563040:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000100:00000200:2.0:1547251557.563047:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b5400 x1622384012442816/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251557.563053:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b5400 x1622384012442816/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251557.563057:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00010000:00080000:2.0:1547251557.563060:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b5400 x1622384012442816/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251557.962688:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442864, portal 4 00000100:00000200:9.0:1547251557.962690:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442864, offset 0 00000400:00000200:9.0:1547251557.962693:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251557.962696:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251557.962698:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03cf0 00000400:00000200:9.0:1547251557.962701:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd3ff9 [1] + 0 00000400:00000200:9.0:1547251557.962703:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ae6a914000 00000100:00000200:9.0:1547251557.962705:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251557.962713:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914000 00000400:00000200:9.0:1547251557.962714:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251557.962717:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf5700 x1622384012442864/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251557.962719:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442880, portal 4 00000400:00000200:9.0:1547251557.962721:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea000 00000400:00000200:9.0:1547251557.962722:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:6.0:1547251557.962722:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442880, offset 0 00000400:00000200:9.0:1547251557.962723:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251557.962724:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251557.962727:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251557.962730:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03d00 00000400:00000200:6.0:1547251557.962733:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1ad5 [1] + 0 00000400:00000200:6.0:1547251557.962735:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040b660 00000100:00000200:6.0:1547251557.962737:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:11.0:1547251557.962744:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442864 00000400:00000200:6.0:1547251557.962746:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b660 00000400:00000200:6.0:1547251557.962747:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251557.962750:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37931b00 x1622384012442880/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00010000:00000200:11.0:1547251557.962753:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3de050 x1622384012442864/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:618/0 lens 224/368 e 0 to 0 dl 1547251563 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:6.0:1547251557.962754:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990bb0 00000400:00000200:6.0:1547251557.962756:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251557.962757:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:11.0:1547251557.962760:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442864, offset 192 00000400:00000200:11.0:1547251557.962763:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:11.0:1547251557.962766:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:11.0:1547251557.962769:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03cf0 00000100:00000200:4.0:1547251557.962769:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442880 00000400:00000200:11.0:1547251557.962771:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae991 [1] + 192 00000100:00000200:11.0:1547251557.962774:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf5700 x1622384012442864/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00000200:4.0:1547251557.962778:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c119c50 x1622384012442880/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:618/0 lens 224/368 e 0 to 0 dl 1547251563 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:11.0:1547251557.962784:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251557.962785:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442880, offset 192 00000400:00000200:11.0:1547251557.962786:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0baa0 00000400:00000200:11.0:1547251557.962786:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251557.962787:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:11.0:1547251557.962788:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251557.962790:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:9.0:1547251557.962792:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf5700 x1622384012442864/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251557.962793:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:4.0:1547251557.962794:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03d00 00000100:00000200:9.0:1547251557.962796:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf5700 x1622384012442864/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251557.962800:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaa18 00000400:00000200:4.0:1547251557.962802:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae9a1 [1] + 192 00010000:00080000:9.0:1547251557.962803:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf5700 x1622384012442864/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251557.962806:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37931b00 x1622384012442880/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251557.962814:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251557.962816:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b990 00000400:00000200:4.0:1547251557.962817:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251557.962818:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251557.962822:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37931b00 x1622384012442880/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251557.962823:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251557.962827:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37931b00 x1622384012442880/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251557.962831:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459902a8 00010000:00080000:6.0:1547251557.962834:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37931b00 x1622384012442880/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251557.978693:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442896, portal 4 00000100:00000200:6.0:1547251557.978696:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442896, offset 0 00000400:00000200:6.0:1547251557.978699:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251557.978702:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251557.978705:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03d10 00000400:00000200:6.0:1547251557.978707:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1b05 [1] + 0 00000400:00000200:6.0:1547251557.978710:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073adcc0 00000100:00000200:6.0:1547251557.978711:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251557.978719:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adcc0 00000400:00000200:6.0:1547251557.978720:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251557.978722:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37937800 x1622384012442896/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251557.978727:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990bb0 00000400:00000200:6.0:1547251557.978727:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251557.978728:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251557.978751:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012442896 00010000:00000200:5.0:1547251557.978760:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c119850 x1622384012442896/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:618/0 lens 224/368 e 0 to 0 dl 1547251563 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251557.978767:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012442896, offset 192 00000400:00000200:5.0:1547251557.978769:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251557.978773:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251557.978775:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03d10 00000400:00000200:5.0:1547251557.978778:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae9b1 [1] + 192 00000100:00000200:5.0:1547251557.978781:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37937800 x1622384012442896/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251557.978791:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251557.978793:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adc38 00000400:00000200:5.0:1547251557.978794:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251557.978795:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251557.978799:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37937800 x1622384012442896/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251557.978800:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251557.978803:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37937800 x1622384012442896/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251557.978807:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459902a8 00010000:00080000:6.0:1547251557.978810:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37937800 x1622384012442896/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251564 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251559.994674:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442912, portal 4 00000100:00000200:8.0:1547251559.994677:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442912, offset 0 00000400:00000200:8.0:1547251559.994680:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:8.0:1547251559.994683:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:8.0:1547251559.994685:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22987] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251559.994688:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22987] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22987] 00000400:00000200:8.0:1547251559.994692:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251559.994694:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251559.994695:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251559.994696:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9724, 9725] 00000400:00000200:8.0:1547251559.994698:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:8.0:1547251559.994703:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251559.994706:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:8.0:1547251559.994708:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251559.994710:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:8.0:1547251559.994711:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:15.0:1547251559.994712:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442928, portal 4 00000800:00000200:8.0:1547251559.994713:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:15.0:1547251559.994715:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442928, offset 0 00000400:00000200:15.0:1547251559.994718:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:15.0:1547251559.994721:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:15.0:1547251559.994724:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22988] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251559.994727:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22987] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22988] 00000400:00000200:15.0:1547251559.994729:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:15.0:1547251559.994731:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251559.994732:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251559.994734:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9725, 9725] 00000400:00000200:15.0:1547251559.994736:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:15.0:1547251559.994740:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251559.994743:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:15.0:1547251559.994746:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251559.994747:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:15.0:1547251559.994748:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251559.994751:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:2.0:1547251559.994760:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442944, portal 4 00000100:00000200:2.0:1547251559.994763:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442944, offset 0 00000800:00000200:9.2:1547251559.994766:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:2.0:1547251559.994766:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:2.0:1547251559.994769:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:2.0:1547251559.994772:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22988] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251559.994775:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22988] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22988] 00000400:00000200:2.0:1547251559.994777:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:2.0:1547251559.994778:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251559.994780:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:7.2:1547251559.994781:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:2.0:1547251559.994781:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9725, 9726] 00000400:00000200:2.0:1547251559.994783:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:2.0:1547251559.994788:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251559.994791:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:13.0:1547251559.994793:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:2.0:1547251559.994794:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251559.994795:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:2.0:1547251559.994796:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (23)++ 00000800:00000200:13.0:1547251559.994797:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:12.0:1547251559.994799:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (24)++ 00000800:00000200:2.0:1547251559.994799:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (24)-- 00000800:00000200:12.0:1547251559.994801:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (24)-- 00000100:00000200:13.0:1547251559.994802:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8caec00 x1622384012442928/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251566 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251559.994805:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824754200 x1622384012442912/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251566 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251559.994809:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1440 00000400:00000200:13.0:1547251559.994811:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251559.994811:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00000400:00000200:12.0:1547251559.994812:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251559.994815:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:12.0:1547251559.994815:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251559.994817:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251559.994819:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:13.0:1547251559.994820:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:12.0:1547251559.994821:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251559.994828:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:13.0:1547251559.994831:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:12.0:1547251559.994831:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f7200 x1622384012442944/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251566 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251559.994837:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a880 00000400:00000200:12.0:1547251559.994838:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251559.994841:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251559.994843:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:9.2:1547251559.994890:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.2:1547251559.994895:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:13.0:1547251559.994902:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:13.0:1547251559.994907:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251559.994909:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:13.0:1547251559.994911:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:9.2:1547251559.994911:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:12.0:1547251559.994913:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.93@o2ib10 00000400:00000200:13.0:1547251559.994917:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03d30 00000400:00000200:12.0:1547251559.994918:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251559.994921:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f79d [1] + 192 00000400:00000200:12.0:1547251559.994922:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03d20 00000400:00000200:12.0:1547251559.994925:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f78d [1] + 192 00000100:00000200:13.0:1547251559.994930:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8caec00 x1622384012442928/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251566 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251559.994950:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:12.0:1547251559.994952:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824754200 x1622384012442912/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251566 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251559.994953:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:13.0:1547251559.994955:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:13.0:1547251559.994956:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:13.0:1547251559.994958:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:13.0:1547251559.994959:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:13.0:1547251559.994961:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:12.0:1547251559.994962:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:15.0:1547251559.994964:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8caec00 x1622384012442928/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251566 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251559.994964:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000800:00000200:12.0:1547251559.994964:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251559.994966:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251559.994967:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000400:00000200:13.0:1547251559.994968:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251559.994969:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:15.0:1547251559.994970:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8caec00 x1622384012442928/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251566 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251559.994972:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03d40 00000400:00000200:15.0:1547251559.994979:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00010000:00080000:15.0:1547251559.994982:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8caec00 x1622384012442928/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251566 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251559.994982:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824754200 x1622384012442912/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251566 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251559.994988:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824754200 x1622384012442912/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251566 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251559.994992:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688088 00000400:00000200:13.0:1547251559.994995:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f7bd [1] + 192 00010000:00080000:8.0:1547251559.994995:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824754200 x1622384012442912/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251566 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:13.0:1547251559.994999:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f7200 x1622384012442944/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251566 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251559.995008:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251559.995009:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:13.0:1547251559.995010:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:13.0:1547251559.995011:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000100:00000200:2.0:1547251559.995017:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f7200 x1622384012442944/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251566 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251559.995023:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f7200 x1622384012442944/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251566 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251559.995027:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00010000:00080000:2.0:1547251559.995030:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f7200 x1622384012442944/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251566 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:15.0:1547251562.106673:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442960, portal 4 00000100:00000200:15.0:1547251562.106676:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442960, offset 0 00000400:00000200:15.0:1547251562.106679:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:15.0:1547251562.106682:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:15.0:1547251562.106684:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22989] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251562.106688:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22988] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22989] 00000400:00000200:15.0:1547251562.106690:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:15.0:1547251562.106692:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251562.106693:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251562.106694:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9730, 9731] 00000400:00000200:15.0:1547251562.106696:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000100:00000200:9.0:1547251562.106697:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442976, portal 4 00000100:00000200:9.0:1547251562.106700:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442976, offset 0 00000400:00000200:15.0:1547251562.106701:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000400:00000200:9.0:1547251562.106703:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:15.0:1547251562.106704:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:9.0:1547251562.106706:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:15.0:1547251562.106707:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251562.106708:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000400:00000200:9.0:1547251562.106708:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22989] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:15.0:1547251562.106709:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:15.0:1547251562.106712:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000400:00000200:9.0:1547251562.106712:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22989] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22989] 00000400:00000200:9.0:1547251562.106714:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251562.106716:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251562.106717:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251562.106718:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9731, 9731] 00000400:00000200:9.0:1547251562.106720:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:9.0:1547251562.106725:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251562.106728:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:9.0:1547251562.106731:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251562.106732:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:9.0:1547251562.106733:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:9.0:1547251562.106736:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000100:00000200:3.0:1547251562.106751:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012442992, portal 4 00000100:00000200:3.0:1547251562.106754:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012442992, offset 0 00000400:00000200:3.0:1547251562.106758:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:3.0:1547251562.106761:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:7.2:1547251562.106762:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000400:00000200:3.0:1547251562.106763:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22990] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251562.106767:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22989] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22990] 00000400:00000200:3.0:1547251562.106769:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:3.0:1547251562.106770:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251562.106772:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251562.106773:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9731, 9732] 00000400:00000200:3.0:1547251562.106775:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:3.0:1547251562.106779:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251562.106782:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:3.0:1547251562.106785:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251562.106787:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251562.106788:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:3.0:1547251562.106788:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (23)++ 00000800:00000200:3.0:1547251562.106790:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (25)-- 00000800:00000200:10.0:1547251562.106796:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000800:00000200:5.2:1547251562.106796:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000100:00000200:10.0:1547251562.106802:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cac200 x1622384012442960/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251562.106803:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:11.0:1547251562.106806:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000100:00000200:11.0:1547251562.106809:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c5100 x1622384012442992/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251562.106810:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1908 00000400:00000200:10.0:1547251562.106811:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251562.106815:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251562.106816:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555088 00000400:00000200:11.0:1547251562.106817:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251562.106818:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000400:00000200:11.0:1547251562.106820:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251562.106820:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251562.106822:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251562.106822:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251562.106824:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:10.0:1547251562.106825:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf3f00 x1622384012442976/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251562.106826:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251562.106831:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea000 00000400:00000200:10.0:1547251562.106832:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251562.106835:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251562.106837:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:5.2:1547251562.106886:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251562.106893:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:7.2:1547251562.106898:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:11.0:1547251562.106901:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251562.106905:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000800:00000200:10.0:1547251562.106906:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251562.106909:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251562.106910:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251562.106913:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251562.106915:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03d50 00000400:00000200:10.0:1547251562.106917:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03d60 00000400:00000200:11.0:1547251562.106919:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd66f9 [1] + 192 00000400:00000200:10.0:1547251562.106924:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6709 [1] + 192 00000100:00000200:11.0:1547251562.106928:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cac200 x1622384012442960/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251562.106947:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251562.106948:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf3f00 x1622384012442976/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251562.106950:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251562.106953:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251562.106954:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251562.106956:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251562.106958:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251562.106959:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:15.0:1547251562.106961:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cac200 x1622384012442960/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251562.106961:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251562.106962:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000400:00000200:11.0:1547251562.106963:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251562.106964:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251562.106965:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000400:00000200:11.0:1547251562.106966:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03d70 00000100:00000200:15.0:1547251562.106967:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cac200 x1622384012442960/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251562.106968:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:10.0:1547251562.106970:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:15.0:1547251562.106971:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00010000:00080000:15.0:1547251562.106974:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cac200 x1622384012442960/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251562.106974:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf3f00 x1622384012442976/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251562.106979:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf3f00 x1622384012442976/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251562.106983:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaa18 00000400:00000200:11.0:1547251562.106987:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6729 [1] + 192 00010000:00080000:9.0:1547251562.106987:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf3f00 x1622384012442976/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251562.106991:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c5100 x1622384012442992/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251562.106999:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251562.107000:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251562.107001:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (22)-- 00000800:00000200:11.0:1547251562.107002:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:3.0:1547251562.107008:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c5100 x1622384012442992/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251562.107013:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c5100 x1622384012442992/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251562.107018:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251562.107021:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c5100 x1622384012442992/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251562.570681:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443008, portal 4 00000100:00000200:8.0:1547251562.570684:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443008, offset 0 00000400:00000200:8.0:1547251562.570687:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251562.570690:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251562.570692:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22990] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251562.570695:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22990] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22990] 00000400:00000200:8.0:1547251562.570698:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000100:00000200:3.0:1547251562.570698:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443024, portal 4 00000400:00000200:8.0:1547251562.570699:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251562.570701:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:3.0:1547251562.570701:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443024, offset 0 00000400:00000200:8.0:1547251562.570702:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9724, 9724] 00000400:00000200:8.0:1547251562.570704:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:3.0:1547251562.570704:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:3.0:1547251562.570707:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251562.570709:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:3.0:1547251562.570709:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22991] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251562.570712:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22990] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22991] 00000800:00000200:8.0:1547251562.570714:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:3.0:1547251562.570715:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:3.0:1547251562.570716:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251562.570717:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251562.570718:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:8.0:1547251562.570719:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000400:00000200:3.0:1547251562.570719:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251562.570720:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9724, 9725] 00000800:00000200:8.0:1547251562.570722:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000400:00000200:3.0:1547251562.570722:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:3.0:1547251562.570727:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251562.570730:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:12.2:1547251562.570732:0:12684:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:3.0:1547251562.570733:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251562.570734:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:3.0:1547251562.570736:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000100:00000200:13.0:1547251562.570738:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443040, portal 4 00000800:00000200:3.0:1547251562.570738:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:13.0:1547251562.570742:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443040, offset 0 00000400:00000200:13.0:1547251562.570746:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:13.0:1547251562.570749:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:13.0:1547251562.570753:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22991] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:0.2:1547251562.570754:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:13.0:1547251562.570757:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22991] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22991] 00000400:00000200:13.0:1547251562.570761:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:13.0:1547251562.570763:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251562.570765:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251562.570767:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9725, 9725] 00000400:00000200:13.0:1547251562.570768:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:13.0:1547251562.570773:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251562.570776:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:13.0:1547251562.570779:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251562.570781:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (22)++ 00000800:00000200:13.0:1547251562.570782:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00000800:00000200:13.0:1547251562.570785:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (24)-- 00000800:00000200:5.0:1547251562.570791:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:5.0:1547251562.570795:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000800:00000200:4.0:1547251562.570795:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251562.570797:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:5.0:1547251562.570799:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824757800 x1622384012443008/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251562.570801:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c4e00 x1622384012443024/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251562.570807:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000400:00000200:4.0:1547251562.570807:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555550 00000400:00000200:5.0:1547251562.570808:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251562.570808:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251562.570812:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251562.570812:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251562.570815:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251562.570816:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251562.570817:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251562.570818:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:4.0:1547251562.570819:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251562.570820:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:4.0:1547251562.570821:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cabc00 x1622384012443040/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251562.570827:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b3b8 00000400:00000200:4.0:1547251562.570829:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251562.570832:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251562.570833:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:0.2:1547251562.570896:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251562.570901:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:0.2:1547251562.570907:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:5.0:1547251562.570910:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251562.570914:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251562.570915:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251562.570918:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251562.570919:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251562.570922:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251562.570924:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03d90 00000400:00000200:4.0:1547251562.570926:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03d80 00000400:00000200:5.0:1547251562.570928:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3465 [1] + 192 00000400:00000200:4.0:1547251562.570933:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3455 [1] + 192 00000100:00000200:5.0:1547251562.570946:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c4e00 x1622384012443024/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251562.570958:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251562.570960:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824757800 x1622384012443008/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251562.570961:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251562.570963:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251562.570964:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251562.570966:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251562.570968:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251562.570971:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251562.570973:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000100:00000200:3.0:1547251562.570973:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c4e00 x1622384012443024/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251562.570974:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000400:00000200:5.0:1547251562.570976:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03da0 00000800:00000200:4.0:1547251562.570976:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251562.570978:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251562.570979:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:3.0:1547251562.570979:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c4e00 x1622384012443024/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251562.570981:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:4.0:1547251562.570982:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:3.0:1547251562.570983:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00000400:00000200:5.0:1547251562.570986:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3475 [1] + 192 00010000:00080000:3.0:1547251562.570987:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c4e00 x1622384012443024/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251562.570991:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824757800 x1622384012443008/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251562.570997:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824757800 x1622384012443008/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251562.571001:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688088 00000100:00000200:5.0:1547251562.571004:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cabc00 x1622384012443040/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251562.571005:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824757800 x1622384012443008/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251562.571012:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251562.571013:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251562.571014:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251562.571015:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000100:00000200:13.0:1547251562.571022:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cabc00 x1622384012443040/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:13.0:1547251562.571028:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cabc00 x1622384012443040/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251562.571032:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792bd48 00010000:00080000:13.0:1547251562.571037:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cabc00 x1622384012443040/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251562.970673:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443056, portal 4 00000100:00000200:8.0:1547251562.970676:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443056, offset 0 00000400:00000200:8.0:1547251562.970679:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251562.970682:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251562.970684:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03db0 00000400:00000200:8.0:1547251562.970687:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4079 [1] + 0 00000400:00000200:8.0:1547251562.970689:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ba9d367d48 00000100:00000200:8.0:1547251562.970691:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:7.0:1547251562.970697:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443072, portal 4 00000400:00000200:8.0:1547251562.970699:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba9d367d48 00000400:00000200:8.0:1547251562.970701:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251562.970701:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443072, offset 0 00000400:00000200:7.0:1547251562.970703:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:8.0:1547251562.970705:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824756300 x1622384012443056/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251562.970706:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251562.970710:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03dc0 00000400:00000200:8.0:1547251562.970711:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000400:00000200:8.0:1547251562.970712:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251562.970712:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1b85 [1] + 0 00000400:00000200:8.0:1547251562.970714:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251562.970714:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad908 00000100:00000200:7.0:1547251562.970716:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:9.0:1547251562.970724:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443056 00000400:00000200:7.0:1547251562.970725:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad908 00000400:00000200:7.0:1547251562.970726:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251562.970730:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df2a00 x1622384012443072/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00010000:00000200:9.0:1547251562.970735:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3ddc50 x1622384012443056/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:623/0 lens 224/368 e 0 to 0 dl 1547251568 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:7.0:1547251562.970736:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:7.0:1547251562.970737:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251562.970739:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251562.970741:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443056, offset 192 00000400:00000200:9.0:1547251562.970743:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251562.970747:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:6.0:1547251562.970749:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443072 00000400:00000200:9.0:1547251562.970750:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03db0 00000400:00000200:9.0:1547251562.970752:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xae9f1 [1] + 192 00000100:00000200:9.0:1547251562.970755:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824756300 x1622384012443056/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00000200:6.0:1547251562.970760:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c119450 x1622384012443072/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:623/0 lens 224/368 e 0 to 0 dl 1547251568 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:9.0:1547251562.970763:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251562.970765:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaa18 00000400:00000200:9.0:1547251562.970766:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:6.0:1547251562.970768:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443072, offset 192 00000400:00000200:9.0:1547251562.970769:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251562.970770:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251562.970772:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:8.0:1547251562.970773:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824756300 x1622384012443056/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251562.970775:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03dc0 00000100:00000200:9.0:1547251562.970776:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:8.0:1547251562.970779:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824756300 x1622384012443056/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251562.970785:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688088 00000400:00000200:6.0:1547251562.970788:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaea01 [1] + 192 00010000:00080000:8.0:1547251562.970790:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824756300 x1622384012443056/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251562.970791:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df2a00 x1622384012443072/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251562.970799:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251562.970801:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459902a8 00000400:00000200:6.0:1547251562.970802:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251562.970804:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251562.970810:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df2a00 x1622384012443072/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251562.970812:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251562.970816:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df2a00 x1622384012443072/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251562.970821:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00010000:00080000:7.0:1547251562.970825:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df2a00 x1622384012443072/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251562.986680:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443088, portal 4 00000100:00000200:7.0:1547251562.986683:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443088, offset 0 00000400:00000200:7.0:1547251562.986685:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251562.986688:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251562.986691:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03dd0 00000400:00000200:7.0:1547251562.986694:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1bb5 [1] + 0 00000400:00000200:7.0:1547251562.986696:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad4c8 00000100:00000200:7.0:1547251562.986698:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:7.0:1547251562.986706:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad4c8 00000400:00000200:7.0:1547251562.986707:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251562.986711:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df2a00 x1622384012443088/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251562.986719:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:7.0:1547251562.986720:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251562.986722:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251562.986729:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443088 00010000:00000200:6.0:1547251562.986739:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c119050 x1622384012443088/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:623/0 lens 224/368 e 0 to 0 dl 1547251568 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251562.986746:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443088, offset 192 00000400:00000200:6.0:1547251562.986748:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251562.986751:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251562.986753:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03dd0 00000400:00000200:6.0:1547251562.986755:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaea11 [1] + 192 00000100:00000200:6.0:1547251562.986758:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df2a00 x1622384012443088/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251562.986767:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251562.986769:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990bb0 00000400:00000200:6.0:1547251562.986770:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251562.986772:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251562.986777:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df2a00 x1622384012443088/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251562.986779:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251562.986783:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df2a00 x1622384012443088/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251562.986788:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00010000:00080000:7.0:1547251562.986792:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df2a00 x1622384012443088/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251569 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:0.0:1547251565.002707:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443104, portal 4 00000100:00000200:0.0:1547251565.002711:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443104, offset 0 00000100:00000200:9.0:1547251565.002715:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443136, portal 4 00000400:00000200:0.0:1547251565.002715:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000100:00000200:9.0:1547251565.002718:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443136, offset 0 00000400:00000200:0.0:1547251565.002718:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:9.0:1547251565.002721:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:0.0:1547251565.002722:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22992] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251565.002724:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:9.0:1547251565.002726:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22992] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251565.002726:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22991] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22992] 00000400:00000200:0.0:1547251565.002729:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251565.002730:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22991] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22992] 00000400:00000200:9.0:1547251565.002732:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251565.002734:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251565.002735:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251565.002736:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9726, 9726] 00000400:00000200:9.0:1547251565.002738:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:9.0:1547251565.002743:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000400:00000200:0.0:1547251565.002744:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251565.002745:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000100:00000200:13.0:1547251565.002747:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443120, portal 4 00000400:00000200:0.0:1547251565.002747:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251565.002748:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000400:00000200:0.0:1547251565.002749:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9726, 9727] 00000800:00000200:9.0:1547251565.002750:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000100:00000200:13.0:1547251565.002751:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443120, offset 0 00000800:00000200:9.0:1547251565.002751:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:9.0:1547251565.002754:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000400:00000200:13.0:1547251565.002755:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:13.0:1547251565.002759:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:13.0:1547251565.002762:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22993] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:13.0:1547251565.002765:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22992] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22993] 00000400:00000200:0.0:1547251565.002765:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:13.0:1547251565.002767:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:13.0:1547251565.002769:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251565.002770:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251565.002771:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9727, 9727] 00000400:00000200:13.0:1547251565.002773:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:13.0:1547251565.002778:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251565.002780:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:7.2:1547251565.002804:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000400:00000200:0.0:1547251565.002805:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251565.002808:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:13.0:1547251565.002809:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251565.002811:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:13.0:1547251565.002812:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:13.0:1547251565.002815:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251565.002824:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:0.0:1547251565.002827:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251565.002834:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000800:00000200:9.2:1547251565.002835:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000100:00000200:12.0:1547251565.002840:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf3c00 x1622384012443136/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251572 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251565.002842:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:13.0:1547251565.002844:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:13.0:1547251565.002846:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000400:00000200:12.0:1547251565.002847:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea5d8 00000400:00000200:12.0:1547251565.002848:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:13.0:1547251565.002849:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cae000 x1622384012443120/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251572 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251565.002852:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:13.0:1547251565.002855:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b2a8 00000800:00000200:12.0:1547251565.002855:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000400:00000200:13.0:1547251565.002856:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:12.0:1547251565.002857:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:13.0:1547251565.002859:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251565.002862:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:0.0:1547251565.002867:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:0.0:1547251565.002905:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:0.0:1547251565.002908:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:9.2:1547251565.002919:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251565.002932:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251565.002945:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000100:00000200:12.0:1547251565.002949:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c4b00 x1622384012443104/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251572 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251565.002955:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:12.0:1547251565.002955:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665440 00000400:00000200:12.0:1547251565.002956:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251565.002959:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251565.002961:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:7.2:1547251565.002970:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251565.002975:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:7.2:1547251565.002979:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:13.0:1547251565.002982:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251565.002983:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:13.0:1547251565.002985:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251565.002986:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.93@o2ib10 00000400:00000200:13.0:1547251565.002990:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251565.002990:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251565.002995:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03e00 00000400:00000200:12.0:1547251565.002995:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03de0 00000400:00000200:13.0:1547251565.002999:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f7fd [1] + 192 00000400:00000200:12.0:1547251565.003003:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f7ed [1] + 192 00000100:00000200:13.0:1547251565.003007:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf3c00 x1622384012443136/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251572 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251565.003018:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:12.0:1547251565.003019:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c4b00 x1622384012443104/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251572 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251565.003020:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000800:00000200:13.0:1547251565.003022:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:13.0:1547251565.003023:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:13.0:1547251565.003026:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:13.0:1547251565.003028:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251565.003029:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251565.003031:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000400:00000200:13.0:1547251565.003032:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000100:00000200:9.0:1547251565.003032:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf3c00 x1622384012443136/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251572 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251565.003033:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251565.003034:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:13.0:1547251565.003036:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03df0 00000800:00000200:12.0:1547251565.003036:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:9.0:1547251565.003037:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf3c00 x1622384012443136/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251572 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251565.003038:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:12.0:1547251565.003039:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:9.0:1547251565.003041:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea2a8 00000400:00000200:13.0:1547251565.003046:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f80d [1] + 192 00010000:00080000:9.0:1547251565.003046:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf3c00 x1622384012443136/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251572 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:0.0:1547251565.003051:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c4b00 x1622384012443104/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251572 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251565.003057:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c4b00 x1622384012443104/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251572 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251565.003061:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00000100:00000200:13.0:1547251565.003063:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cae000 x1622384012443120/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251572 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251565.003064:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c4b00 x1622384012443104/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251572 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:13.0:1547251565.003074:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251565.003075:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:13.0:1547251565.003076:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:13.0:1547251565.003077:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000100:00000200:14.0:1547251565.003121:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cae000 x1622384012443120/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251572 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:14.0:1547251565.003127:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cae000 x1622384012443120/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251572 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251565.003133:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792bd48 00010000:00080000:14.0:1547251565.003137:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cae000 x1622384012443120/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251572 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251567.114686:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443152, portal 4 00000100:00000200:8.0:1547251567.114688:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443152, offset 0 00000400:00000200:8.0:1547251567.114691:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251567.114694:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251567.114697:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22993] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251567.114700:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22993] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22993] 00000400:00000200:8.0:1547251567.114702:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251567.114704:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251567.114705:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251567.114706:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9732, 9732] 00000400:00000200:8.0:1547251567.114708:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:8.0:1547251567.114713:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251567.114715:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:8.0:1547251567.114718:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000100:00000200:0.0:1547251567.114719:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443168, portal 4 00000100:00000200:15.0:1547251567.114720:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443184, portal 4 00000800:00000200:8.0:1547251567.114720:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:8.0:1547251567.114721:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000100:00000200:15.0:1547251567.114723:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443184, offset 0 00000800:00000200:8.0:1547251567.114723:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000100:00000200:0.0:1547251567.114723:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443168, offset 0 00000400:00000200:15.0:1547251567.114727:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:0.0:1547251567.114727:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:15.0:1547251567.114730:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:0.0:1547251567.114730:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:15.0:1547251567.114732:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22994] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251567.114732:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:22994] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251567.114735:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22993] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22994] 00000400:00000200:15.0:1547251567.114736:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22993] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:22994] 00000400:00000200:15.0:1547251567.114738:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251567.114738:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251567.114739:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251567.114740:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251567.114741:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251567.114741:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251567.114742:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9732, 9733] 00000400:00000200:15.0:1547251567.114743:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9732, 9733] 00000400:00000200:0.0:1547251567.114744:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:15.0:1547251567.114745:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:0.0:1547251567.114749:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251567.114751:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:15.0:1547251567.114754:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251567.114754:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251567.114756:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:15.0:1547251567.114757:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:0.0:1547251567.114757:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:0.0:1547251567.114759:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000800:00000200:15.0:1547251567.114760:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251567.114762:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (21)++ 00000800:00000200:15.0:1547251567.114763:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (22)++ 00000800:00000200:15.0:1547251567.114766:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (23)-- 00000800:00000200:5.2:1547251567.114774:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251567.114799:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251567.114808:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000800:00000200:7.2:1547251567.114808:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (22)++ 00000100:00000200:10.0:1547251567.114813:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824755400 x1622384012443152/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251567.114815:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251567.114818:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:11.0:1547251567.114820:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000400:00000200:10.0:1547251567.114820:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000400:00000200:10.0:1547251567.114822:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:11.0:1547251567.114823:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f6f00 x1622384012443168/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251567.114825:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251567.114828:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000400:00000200:11.0:1547251567.114830:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665198 00000800:00000200:10.0:1547251567.114830:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000400:00000200:11.0:1547251567.114831:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251567.114831:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000400:00000200:11.0:1547251567.114834:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000100:00000200:10.0:1547251567.114834:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cae000 x1622384012443184/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251567.114836:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251567.114838:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:10.0:1547251567.114840:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1550 00000400:00000200:10.0:1547251567.114841:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251567.114844:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251567.114846:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:5.2:1547251567.114903:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251567.114909:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:11.0:1547251567.114916:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251567.114920:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000800:00000200:10.0:1547251567.114923:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:11.0:1547251567.114924:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251567.114925:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251567.114929:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03e20 00000400:00000200:10.0:1547251567.114931:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251567.114934:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd67a9 [1] + 192 00000400:00000200:10.0:1547251567.114934:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03e10 00000400:00000200:10.0:1547251567.114947:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6789 [1] + 192 00000100:00000200:11.0:1547251567.114951:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f6f00 x1622384012443168/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251567.114963:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251567.114964:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824755400 x1622384012443152/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251567.114966:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251567.114968:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251567.114970:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251567.114971:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251567.114974:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251567.114975:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251567.114977:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000100:00000200:0.0:1547251567.114977:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f6f00 x1622384012443168/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251567.114978:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251567.114978:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251567.114980:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:11.0:1547251567.114981:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03e30 00000800:00000200:10.0:1547251567.114982:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:10.0:1547251567.114983:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000100:00000200:0.0:1547251567.114983:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f6f00 x1622384012443168/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251567.114992:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00000100:00000200:8.0:1547251567.114995:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824755400 x1622384012443152/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251567.114995:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f6f00 x1622384012443168/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251567.115001:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824755400 x1622384012443152/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251567.115005:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688088 00000400:00000200:11.0:1547251567.115008:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd67b9 [1] + 192 00010000:00080000:8.0:1547251567.115009:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824755400 x1622384012443152/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251567.115011:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cae000 x1622384012443184/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251567.115018:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251567.115019:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251567.115021:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251567.115021:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:15.0:1547251567.115028:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cae000 x1622384012443184/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:15.0:1547251567.115033:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cae000 x1622384012443184/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251567.115037:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00010000:00080000:15.0:1547251567.115041:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cae000 x1622384012443184/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251567.578715:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443216, portal 4 00000100:00000200:9.0:1547251567.578718:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443216, offset 0 00000100:00000200:0.0:1547251567.578718:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443200, portal 4 00000100:00000200:12.0:1547251567.578720:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443232, portal 4 00000400:00000200:9.0:1547251567.578721:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000100:00000200:0.0:1547251567.578722:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443200, offset 0 00000100:00000200:12.0:1547251567.578723:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443232, offset 0 00000400:00000200:9.0:1547251567.578724:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:12.0:1547251567.578726:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251567.578726:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22994] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251567.578726:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251567.578729:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22995] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22994] 00000400:00000200:12.0:1547251567.578730:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:0.0:1547251567.578730:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:12.0:1547251567.578732:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22994] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251567.578732:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:0.0:1547251567.578732:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22994] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251567.578733:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251567.578735:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22995] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22995] 00000400:00000200:9.0:1547251567.578735:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251567.578735:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22995] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22995] 00000400:00000200:9.0:1547251567.578736:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9725, 9726] 00000400:00000200:12.0:1547251567.578738:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251567.578738:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:0.0:1547251567.578738:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:12.0:1547251567.578739:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251567.578739:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251567.578741:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251567.578741:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251567.578743:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9726, 9726] 00000400:00000200:0.0:1547251567.578743:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9726, 9726] 00000400:00000200:9.0:1547251567.578744:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:12.0:1547251567.578745:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:0.0:1547251567.578745:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:9.0:1547251567.578746:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:9.0:1547251567.578750:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:0.0:1547251567.578750:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251567.578751:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:9.0:1547251567.578752:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:0.0:1547251567.578752:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:12.0:1547251567.578755:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251567.578755:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:0.0:1547251567.578756:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251567.578757:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:12.0:1547251567.578758:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:0.0:1547251567.578758:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:0.0:1547251567.578761:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:12.2:1547251567.578763:0:12682:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000800:00000200:12.2:1547251567.578770:0:12682:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:12.0:1547251567.578778:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251567.578780:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (22)++ 00000800:00000200:12.0:1547251567.578781:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00000800:00000200:12.0:1547251567.578783:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (24)-- 00000800:00000200:4.0:1547251567.578823:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:5.0:1547251567.578826:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251567.578826:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000800:00000200:5.0:1547251567.578829:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000100:00000200:4.0:1547251567.578831:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f6c00 x1622384012443200/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251567.578832:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf2d00 x1622384012443216/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251567.578838:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea220 00000400:00000200:4.0:1547251567.578838:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665b28 00000400:00000200:5.0:1547251567.578840:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251567.578840:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251567.578844:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251567.578844:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251567.578846:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251567.578848:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251567.578849:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251567.578850:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:4.0:1547251567.578850:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251567.578851:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:4.0:1547251567.578853:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f0300 x1622384012443232/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251567.578859:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038110 00000400:00000200:4.0:1547251567.578860:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251567.578863:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251567.578865:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:12.2:1547251567.578947:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:0.2:1547251567.578952:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:12.2:1547251567.578961:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:5.0:1547251567.578964:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251567.578968:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251567.578968:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251567.578971:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251567.578973:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251567.578976:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251567.578978:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03e50 00000400:00000200:4.0:1547251567.578979:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03e40 00000400:00000200:5.0:1547251567.578982:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe34e5 [1] + 192 00000400:00000200:4.0:1547251567.578987:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe34f5 [1] + 192 00000100:00000200:5.0:1547251567.578991:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf2d00 x1622384012443216/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251567.579003:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251567.579005:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000100:00000200:4.0:1547251567.579005:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f6c00 x1622384012443200/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251567.579008:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251567.579009:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251567.579011:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251567.579013:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251567.579016:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251567.579017:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000100:00000200:9.0:1547251567.579018:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf2d00 x1622384012443216/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251567.579018:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251567.579019:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000400:00000200:5.0:1547251567.579021:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03e60 00000800:00000200:4.0:1547251567.579021:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000100:00000200:9.0:1547251567.579023:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf2d00 x1622384012443216/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251567.579024:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:4.0:1547251567.579025:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251567.579027:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:9.0:1547251567.579028:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea2a8 00010000:00080000:9.0:1547251567.579031:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf2d00 x1622384012443216/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251567.579031:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3505 [1] + 192 00000100:00000200:0.0:1547251567.579035:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f6c00 x1622384012443200/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251567.579041:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f6c00 x1622384012443200/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251567.579045:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00000100:00000200:5.0:1547251567.579048:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f0300 x1622384012443232/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251567.579049:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f6c00 x1622384012443200/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251567.579056:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251567.579057:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251567.579058:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251567.579059:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:12.0:1547251567.579065:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f0300 x1622384012443232/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251567.579070:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f0300 x1622384012443232/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251567.579074:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038cc0 00010000:00080000:12.0:1547251567.579079:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f0300 x1622384012443232/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:8.0:1547251567.637887:0:15956:0:(o2iblnd_cb.c:2637:kiblnd_passive_connect()) peer_ni[ffff97b971916c00] -> 10.0.10.184@o2ib10 (1)++ 00000800:00000200:8.0:1547251567.637899:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b6d042b000 0x129042b000(0x129042b000) 00000800:00000200:8.0:1547251567.637900:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b8245ce000 0x13e45ce000(0x13e45ce000) 00000800:00000200:8.0:1547251567.637901:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97a6a93d2000 0x2693d2000(0x2693d2000) 00000800:00000200:8.0:1547251567.637903:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97badc928000 0x169c928000(0x169c928000) 00000800:00000200:8.0:1547251567.637904:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ae3b9b7000 0x9fb9b7000(0x9fb9b7000) 00000800:00000200:8.0:1547251567.637905:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97badc890000 0x169c890000(0x169c890000) 00000800:00000200:8.0:1547251567.637906:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97badf785000 0x169f785000(0x169f785000) 00000800:00000200:8.0:1547251567.637907:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba9baf1000 0x165baf1000(0x165baf1000) 00000800:00000200:8.0:1547251567.637908:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97badc8ef000 0x169c8ef000(0x169c8ef000) 00000800:00000200:8.0:1547251567.637909:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b17b152000 0xd3b152000(0xd3b152000) 00000800:00000200:8.0:1547251567.637910:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b17b360000 0xd3b360000(0xd3b360000) 00000800:00000200:8.0:1547251567.637911:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b6d042d000 0x129042d000(0x129042d000) 00000800:00000200:8.0:1547251567.637911:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b17b0e1000 0xd3b0e1000(0xd3b0e1000) 00000800:00000200:8.0:1547251567.637912:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba8b001000 0x164b001000(0x164b001000) 00000800:00000200:8.0:1547251567.637913:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b82459d000 0x13e459d000(0x13e459d000) 00000800:00000200:8.0:1547251567.637914:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b17b1be000 0xd3b1be000(0xd3b1be000) 00000800:00000200:8.0:1547251567.637915:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97badcbbe000 0x169cbbe000(0x169cbbe000) 00000800:00000200:8.0:1547251567.637916:0:15956:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97badc96b000 0x169c96b000(0x169c96b000) 00000800:00000200:8.0:1547251567.638777:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638779:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638779:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638780:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638780:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638781:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638782:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638782:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638783:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638783:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638784:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638785:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638785:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638786:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638787:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638787:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638788:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638789:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638789:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638791:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638791:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638792:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638793:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638793:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638794:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638795:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638795:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638796:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638797:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638797:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638798:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638798:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638799:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638800:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638800:0:15956:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.638801:0:15956:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (20)-- 00000800:00000200:8.0:1547251567.638803:0:15956:0:(o2iblnd_cb.c:2679:kiblnd_passive_connect()) Accept 10.0.10.184@o2ib10 00000800:00000200:8.0:1547251567.639112:0:15956:0:(o2iblnd_cb.c:3141:kiblnd_cm_callback()) connreq: 0 00000800:00000200:8.0:1547251567.639418:0:15956:0:(o2iblnd_cb.c:3258:kiblnd_cm_callback()) ESTABLISHED (passive): 10.0.10.184@o2ib10 00000800:00000200:8.0:1547251567.639420:0:15956:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.0.10.184@o2ib10: active(0), version(12), status(0) 00000800:00000200:8.0:1547251567.639422:0:15956:0:(o2iblnd_cb.c:2308:kiblnd_connreq_done()) conn[ffff97b81d647a00] (19)++ 00000800:00000200:8.0:1547251567.639423:0:15956:0:(o2iblnd_cb.c:2351:kiblnd_connreq_done()) conn[ffff97b81d647a00] (20)++ 00000800:00000200:8.0:1547251567.639424:0:15956:0:(o2iblnd_cb.c:2371:kiblnd_connreq_done()) conn[ffff97b81d647a00] (21)-- 00000800:00000200:0.2:1547251567.639460:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b81d647a00] (20)++ 00000800:00000200:12.0:1547251567.639514:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:12.0:1547251567.639525:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.10.184@o2ib10 00000400:00000200:12.0:1547251567.639531:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.10.184@o2ib10 : GET - for me 00000400:00000200:12.0:1547251567.639536:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.10.184@o2ib10 of length 272 into portal 0 MB=0x8000000000000000 00000800:00000200:13.0:1547251567.639539:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)-- 00000400:00000200:12.0:1547251567.639541:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming get index 0 from 12345-10.0.10.184@o2ib10 of length 64/272 into md 0x1 [1] + 0 00000800:00000200:12.0:1547251567.639545:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:12.0:1547251567.639547:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (22)-- 00000400:00000200:12.0:1547251567.639551:0:12665:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.10.184@o2ib10 local destination 00000400:00000200:12.0:1547251567.639554:0:12665:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251567.639556:0:12665:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251567.639557:0:12665:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.0.10.184@o2ib10 c:[8, 8], s:[15, 16] 00000400:00000200:12.0:1547251567.639559:0:12665:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.10.184@o2ib10 00000400:00000200:12.0:1547251567.639564:0:12665:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.10.184@o2ib10(10.0.10.184@o2ib10:10.0.10.184@o2ib10) : REPLY try# 0 00000800:00000200:12.0:1547251567.639567:0:12665:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 64 bytes in 1 frags to 12345-10.0.10.184@o2ib10 00000800:00000200:12.0:1547251567.639570:0:12665:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b971916c00] -> 10.0.10.184@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251567.639571:0:12665:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:12.0:1547251567.639572:0:12665:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b81d647a00] (22)++ 00000800:00000200:12.0:1547251567.639574:0:12665:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b81d647a00] (23)-- 00000800:00000200:12.0:1547251567.639575:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)-- 00000800:00000200:0.2:1547251567.639587:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:13.0:1547251567.639602:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)++ 00000800:00000200:13.0:1547251567.639609:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b81d647a00] (23)-- 00000400:00000200:13.0:1547251567.639611:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:12.0:1547251567.639612:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)-- 00000400:00000200:13.0:1547251567.639615:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.10.184@o2ib10: REPLY: OK 00000800:00000200:13.0:1547251567.639618:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (21)-- 00000800:00000200:12.0:1547251567.641090:0:70:0:(o2iblnd_cb.c:2637:kiblnd_passive_connect()) peer_ni[ffff97a59c828d80] -> 10.0.10.184@o2ib10 (1)++ 00000800:00000200:12.0:1547251567.641099:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba96307000 0x1656307000(0x1656307000) 00000800:00000200:12.0:1547251567.641100:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b8245e0000 0x13e45e0000(0x13e45e0000) 00000800:00000200:12.0:1547251567.641101:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97badf659000 0x169f659000(0x169f659000) 00000800:00000200:12.0:1547251567.641102:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b81d611000 0x13dd611000(0x13dd611000) 00000800:00000200:12.0:1547251567.641103:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b8244b4000 0x13e44b4000(0x13e44b4000) 00000800:00000200:12.0:1547251567.641104:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ae6a939000 0xa2a939000(0xa2a939000) 00000800:00000200:12.0:1547251567.641104:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b17b148000 0xd3b148000(0xd3b148000) 00000800:00000200:12.0:1547251567.641105:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b17b149000 0xd3b149000(0xd3b149000) 00000800:00000200:12.0:1547251567.641107:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ae37bac000 0x9f7bac000(0x9f7bac000) 00000800:00000200:12.0:1547251567.641108:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ae37bad000 0x9f7bad000(0x9f7bad000) 00000800:00000200:12.0:1547251567.641109:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b17b168000 0xd3b168000(0xd3b168000) 00000800:00000200:12.0:1547251567.641110:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b17b169000 0xd3b169000(0xd3b169000) 00000800:00000200:12.0:1547251567.641111:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b81d634000 0x13dd634000(0x13dd634000) 00000800:00000200:12.0:1547251567.641112:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b81d635000 0x13dd635000(0x13dd635000) 00000800:00000200:12.0:1547251567.641113:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b81d59e000 0x13dd59e000(0x13dd59e000) 00000800:00000200:12.0:1547251567.641113:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b81d59f000 0x13dd59f000(0x13dd59f000) 00000800:00000200:12.0:1547251567.641114:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b17b040000 0xd3b040000(0xd3b040000) 00000800:00000200:12.0:1547251567.641115:0:70:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b17b041000 0xd3b041000(0xd3b041000) 00000800:00000200:12.0:1547251567.641975:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641976:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641977:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641978:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641978:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641979:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641980:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641980:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641981:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641981:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641982:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641983:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641983:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641984:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641985:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641985:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641986:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641987:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641987:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641988:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641988:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641989:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641990:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641990:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641991:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641992:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641993:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641994:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641994:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641995:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641996:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641997:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641997:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641998:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.641999:0:70:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.641999:0:70:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (20)-- 00000800:00000200:12.0:1547251567.642001:0:70:0:(o2iblnd_cb.c:2679:kiblnd_passive_connect()) Accept 10.0.10.184@o2ib10 00000800:00000200:12.0:1547251567.642309:0:70:0:(o2iblnd_cb.c:3141:kiblnd_cm_callback()) connreq: 0 00000800:00000200:12.0:1547251567.642622:0:70:0:(o2iblnd_cb.c:3258:kiblnd_cm_callback()) ESTABLISHED (passive): 10.0.10.184@o2ib10 00000800:00000200:12.0:1547251567.642623:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.0.10.184@o2ib10: active(0), version(12), status(0) 00000800:00000200:12.0:1547251567.642625:0:70:0:(o2iblnd_cb.c:2308:kiblnd_connreq_done()) conn[ffff97a65d02b200] (19)++ 00000800:00000200:12.0:1547251567.642626:0:70:0:(o2iblnd_cb.c:2351:kiblnd_connreq_done()) conn[ffff97a65d02b200] (20)++ 00000800:00000200:12.0:1547251567.642627:0:70:0:(o2iblnd_cb.c:2371:kiblnd_connreq_done()) conn[ffff97a65d02b200] (21)-- 00000800:00000200:8.2:1547251567.642640:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d02b200] (20)++ 00000800:00000200:12.0:1547251567.642654:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b200] (21)++ 00000800:00000200:12.0:1547251567.642661:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.10.184@o2ib10 00000400:00000200:12.0:1547251567.642665:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.10.184@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251567.642671:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.10.184@o2ib10 of length 48 into portal 0 MB=0x8000000000000000 00000800:00000200:13.0:1547251567.642676:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b200] (22)-- 00000400:00000200:12.0:1547251567.642676:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 0 from 12345-10.0.10.184@o2ib10 of length 48/48 into md 0x11 [1] + 0 00000400:00000200:12.0:1547251567.642682:0:12665:0:(peer.c:1955:lnet_peer_push_event()) Stale Push from 10.0.10.184@o2ib10: got 1 have 1 00000400:00000200:12.0:1547251567.642684:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:12.0:1547251567.642688:0:12665:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.10.184@o2ib10 local destination 00000400:00000200:12.0:1547251567.642692:0:12665:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251567.642694:0:12665:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251567.642696:0:12665:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.0.10.184@o2ib10 c:[8, 8], s:[16, 16] 00000400:00000200:12.0:1547251567.642698:0:12665:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.2.10.184@o2ib10 00000400:00000200:12.0:1547251567.642702:0:12665:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.2.10.184@o2ib10(10.0.10.184@o2ib10:10.2.10.184@o2ib10) : ACK try# 0 00000800:00000200:12.0:1547251567.642705:0:12665:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 0 bytes in 1 frags to 12345-10.2.10.184@o2ib10 00000800:00000200:12.0:1547251567.642709:0:12665:0:(o2iblnd_cb.c:1604:kiblnd_launch_tx()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (1)++ 00000800:00000200:12.0:1547251567.642713:0:12665:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:12.0:1547251567.642727:0:12665:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:12.0:1547251567.642729:0:12665:0:(o2iblnd_cb.c:1611:kiblnd_launch_tx()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:12.0:1547251567.642730:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (21)++ 00000800:00000200:12.0:1547251567.642731:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (22)-- 00000800:00000200:12.0:1547251567.642733:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b200] (21)-- 00000800:00000200:14.0:1547251567.642735:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:14.0:1547251567.642747:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:14.0:1547251567.643236:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:14.0:1547251567.643245:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97badc999000 0x169c999000(0x169c999000) 00000800:00000200:14.0:1547251567.643247:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97acce264000 0x88e264000(0x88e264000) 00000800:00000200:14.0:1547251567.643248:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba972ef000 0x16572ef000(0x16572ef000) 00000800:00000200:14.0:1547251567.643249:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ba9eec4000 0x165eec4000(0x165eec4000) 00000800:00000200:14.0:1547251567.643249:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b05474b000 0xc1474b000(0xc1474b000) 00000800:00000200:14.0:1547251567.643250:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b06f306000 0xc2f306000(0xc2f306000) 00000800:00000200:14.0:1547251567.643251:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b62c426000 0x11ec426000(0x11ec426000) 00000800:00000200:14.0:1547251567.643252:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b5872c8000 0x11472c8000(0x11472c8000) 00000800:00000200:14.0:1547251567.643253:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97badc80b000 0x169c80b000(0x169c80b000) 00000800:00000200:14.0:1547251567.643254:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ad4c2d6000 0x90c2d6000(0x90c2d6000) 00000800:00000200:14.0:1547251567.643255:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ba997d1000 0x16597d1000(0x16597d1000) 00000800:00000200:14.0:1547251567.643256:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba9ef48000 0x165ef48000(0x165ef48000) 00000800:00000200:14.0:1547251567.643257:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97baa13a2000 0x16613a2000(0x16613a2000) 00000800:00000200:14.0:1547251567.643257:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba9d3c1000 0x165d3c1000(0x165d3c1000) 00000800:00000200:14.0:1547251567.643258:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba996e3000 0x16596e3000(0x16596e3000) 00000800:00000200:14.0:1547251567.643259:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97bad6ae0000 0x1696ae0000(0x1696ae0000) 00000800:00000200:14.0:1547251567.643260:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ac1b2d3000 0x7db2d3000(0x7db2d3000) 00000800:00000200:14.0:1547251567.643261:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97a589ecb000 0x149ecb000(0x149ecb000) 00000800:00000200:9.0:1547251567.644027:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644029:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644030:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644031:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644031:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644033:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644034:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644034:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644035:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644036:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644036:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644037:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644038:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644038:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644039:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644040:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644040:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644041:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644041:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644042:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644043:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644043:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644044:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644044:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644045:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644046:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644046:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644047:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644047:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644048:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644049:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644049:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644050:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644050:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251567.644051:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251567.644052:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000100:12.0:1547251567.644328:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251567.644332:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251567.644334:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:12.0:1547251567.644385:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97babe8ae800] (19)-- 00000800:00000200:8.2:1547251567.644386:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:13.0:1547251567.644413:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:13.0:1547251567.644420:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:13.0:1547251567.644422:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (19)-- 00000800:00000200:13.0:1547251567.644424:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (18)-- 00000800:00000200:8.2:1547251567.644512:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ae800] (17)++ 00000800:00000200:13.0:1547251567.644524:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (18)++ 00000800:00000200:13.0:1547251567.644528:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (19)-- 00000800:00000200:13.0:1547251567.644529:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (18)-- 00000800:00000200:13.0:1547251567.644533:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (17)++ 00000800:00000200:13.0:1547251567.644534:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (18)-- 00000800:00000200:12.0:1547251567.644534:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (18)++ 00000800:00000200:13.0:1547251567.644535:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (18)-- 00000800:00000200:13.0:1547251567.644536:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (17)++ 00000800:00000200:12.0:1547251567.644536:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (17)-- 00000800:00000200:12.0:1547251567.644537:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (17)-- 00000800:00000200:13.0:1547251567.644538:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (16)-- 00000800:00000200:12.0:1547251567.644539:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (16)++ 00000800:00000200:13.0:1547251567.644540:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (16)-- 00000800:00000200:13.0:1547251567.644541:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (15)++ 00000800:00000200:12.0:1547251567.644541:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (15)-- 00000800:00000200:12.0:1547251567.644542:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (15)-- 00000800:00000200:13.0:1547251567.644543:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (14)-- 00000800:00000200:12.0:1547251567.644543:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (14)++ 00000800:00000200:13.0:1547251567.644544:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (14)-- 00000800:00000200:13.0:1547251567.644545:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (13)++ 00000800:00000200:12.0:1547251567.644545:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (13)-- 00000800:00000200:12.0:1547251567.644546:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (13)-- 00000800:00000200:13.0:1547251567.644547:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (12)-- 00000800:00000200:12.0:1547251567.644547:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (12)++ 00000800:00000200:13.0:1547251567.644548:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (12)-- 00000800:00000200:13.0:1547251567.644549:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (11)++ 00000800:00000200:12.0:1547251567.644549:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (11)-- 00000800:00000200:12.0:1547251567.644550:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (11)-- 00000800:00000200:13.0:1547251567.644551:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (10)-- 00000800:00000200:12.0:1547251567.644551:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (10)++ 00000800:00000200:13.0:1547251567.644552:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (10)-- 00000800:00000200:13.0:1547251567.644553:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (9)++ 00000800:00000200:12.0:1547251567.644553:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (9)-- 00000800:00000200:12.0:1547251567.644554:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (9)-- 00000800:00000200:13.0:1547251567.644555:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (8)-- 00000800:00000200:12.0:1547251567.644555:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (8)++ 00000800:00000200:13.0:1547251567.644556:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (8)-- 00000800:00000200:13.0:1547251567.644557:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (7)++ 00000800:00000200:12.0:1547251567.644557:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (7)-- 00000800:00000200:12.0:1547251567.644558:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (7)-- 00000800:00000200:13.0:1547251567.644559:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (6)-- 00000800:00000200:12.0:1547251567.644559:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (6)++ 00000800:00000200:13.0:1547251567.644560:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (6)-- 00000800:00000200:13.0:1547251567.644561:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (5)++ 00000800:00000200:12.0:1547251567.644561:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (5)-- 00000800:00000200:12.0:1547251567.644562:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (5)-- 00000800:00000200:13.0:1547251567.644563:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (4)-- 00000800:00000200:12.0:1547251567.644563:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (4)++ 00000800:00000200:13.0:1547251567.644564:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (4)-- 00000800:00000200:13.0:1547251567.644565:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (3)-- 00000800:00000200:12.0:1547251567.644565:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (3)-- 00000800:00000200:12.0:1547251567.644566:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (1)-- 00000800:00000200:10.0:1547251567.644618:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.645468:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251567.645474:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.645485:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251567.645487:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:2.0:1547251567.645493:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:2.0:1547251567.645503:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:2.0:1547251567.645749:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:2.0:1547251567.645760:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b4ef7cf000 0x10af7cf000(0x10af7cf000) 00000800:00000200:2.0:1547251567.645761:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ae6ab1a000 0xa2ab1a000(0xa2ab1a000) 00000800:00000200:2.0:1547251567.645762:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ae6ab1b000 0xa2ab1b000(0xa2ab1b000) 00000800:00000200:2.0:1547251567.645763:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b4f05ec000 0x10b05ec000(0x10b05ec000) 00000800:00000200:2.0:1547251567.645764:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b4f05ed000 0x10b05ed000(0x10b05ed000) 00000800:00000200:2.0:1547251567.645765:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b17b186000 0xd3b186000(0xd3b186000) 00000800:00000200:2.0:1547251567.645766:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b17b187000 0xd3b187000(0xd3b187000) 00000800:00000200:2.0:1547251567.645767:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b6ed5a4000 0x12ad5a4000(0x12ad5a4000) 00000800:00000200:2.0:1547251567.645768:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b6ed5a5000 0x12ad5a5000(0x12ad5a5000) 00000800:00000200:2.0:1547251567.645769:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba99e40000 0x1659e40000(0x1659e40000) 00000800:00000200:2.0:1547251567.645770:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b4f05e0000 0x10b05e0000(0x10b05e0000) 00000800:00000200:2.0:1547251567.645770:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b907301000 0x14c7301000(0x14c7301000) 00000800:00000200:2.0:1547251567.645771:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97badfab2000 0x169fab2000(0x169fab2000) 00000800:00000200:2.0:1547251567.645772:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97badcc4a000 0x169cc4a000(0x169cc4a000) 00000800:00000200:2.0:1547251567.645773:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97bade651000 0x169e651000(0x169e651000) 00000800:00000200:2.0:1547251567.645774:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba96fb2000 0x1656fb2000(0x1656fb2000) 00000800:00000200:2.0:1547251567.645775:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ba95254000 0x1655254000(0x1655254000) 00000800:00000200:2.0:1547251567.645775:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97bae1be9000 0x16a1be9000(0x16a1be9000) 00000800:00000200:9.0:1547251567.646468:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646470:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646471:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646471:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646472:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646473:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646473:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646474:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646474:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646475:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646476:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646476:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646477:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646477:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646478:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646479:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646479:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646480:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646480:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646481:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646482:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646482:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646483:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646483:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646484:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646485:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646485:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646486:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646486:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646487:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646488:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646488:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646489:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646489:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000200:9.0:1547251567.646490:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583a000] (19)++ 00000800:00000200:9.0:1547251567.646491:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583a000] (20)-- 00000800:00000100:12.0:1547251567.646737:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251567.646741:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251567.646743:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251567.646781:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b82583a000] (19)++ 00000800:00000200:13.0:1547251567.646800:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (20)++ 00000800:00000200:13.0:1547251567.646805:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (21)-- 00000800:00000200:13.0:1547251567.646806:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (20)-- 00000800:00000200:13.0:1547251567.646808:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (19)-- 00000800:00000200:12.0:1547251567.646819:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97b82583a000] (18)-- 00000800:00000200:8.2:1547251567.646930:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b82583a000] (17)++ 00000800:00000200:13.0:1547251567.646946:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (18)++ 00000800:00000200:13.0:1547251567.646950:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (19)-- 00000800:00000200:13.0:1547251567.646951:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (18)-- 00000800:00000200:13.0:1547251567.646952:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (17)++ 00000800:00000200:13.0:1547251567.646953:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (18)-- 00000800:00000200:13.0:1547251567.646954:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (17)-- 00000800:00000200:13.0:1547251567.646956:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (16)++ 00000800:00000200:13.0:1547251567.646957:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (17)-- 00000800:00000200:12.0:1547251567.646958:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (17)++ 00000800:00000200:12.0:1547251567.646959:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (18)-- 00000800:00000200:13.0:1547251567.646960:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (17)-- 00000800:00000200:12.0:1547251567.646961:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (15)-- 00000800:00000200:13.0:1547251567.646962:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (14)++ 00000800:00000200:13.0:1547251567.646965:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (15)-- 00000800:00000200:13.0:1547251567.646966:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (14)-- 00000800:00000200:13.0:1547251567.646968:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (13)++ 00000800:00000200:13.0:1547251567.646971:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (14)-- 00000800:00000200:13.0:1547251567.646972:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (13)-- 00000800:00000200:13.0:1547251567.646973:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (12)++ 00000800:00000200:13.0:1547251567.646977:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (13)-- 00000800:00000200:13.0:1547251567.646978:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (12)-- 00000800:00000200:13.0:1547251567.646979:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (11)++ 00000800:00000200:13.0:1547251567.646982:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (12)-- 00000800:00000200:13.0:1547251567.646983:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (11)-- 00000800:00000200:13.0:1547251567.646984:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (10)++ 00000800:00000200:13.0:1547251567.646987:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (11)-- 00000800:00000200:13.0:1547251567.646988:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (10)-- 00000800:00000200:13.0:1547251567.646990:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (9)++ 00000800:00000200:13.0:1547251567.646993:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (10)-- 00000800:00000200:13.0:1547251567.646994:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (9)-- 00000800:00000200:13.0:1547251567.646995:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (8)++ 00000800:00000200:13.0:1547251567.646998:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (9)-- 00000800:00000200:13.0:1547251567.646999:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (8)-- 00000800:00000200:13.0:1547251567.647000:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (7)++ 00000800:00000200:13.0:1547251567.647004:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (8)-- 00000800:00000200:13.0:1547251567.647005:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (7)-- 00000800:00000200:13.0:1547251567.647006:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (6)++ 00000800:00000200:13.0:1547251567.647009:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (7)-- 00000800:00000200:13.0:1547251567.647010:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (6)-- 00000800:00000200:13.0:1547251567.647011:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (5)++ 00000800:00000200:13.0:1547251567.647014:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (6)-- 00000800:00000200:13.0:1547251567.647015:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (5)-- 00000800:00000200:13.0:1547251567.647016:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (4)++ 00000800:00000200:13.0:1547251567.647020:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (5)-- 00000800:00000200:13.0:1547251567.647020:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (4)-- 00000800:00000200:13.0:1547251567.647022:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (3)++ 00000800:00000200:13.0:1547251567.647025:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (4)-- 00000800:00000200:13.0:1547251567.647026:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (3)-- 00000800:00000200:13.0:1547251567.647028:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583a000] (2)++ 00000800:00000200:13.0:1547251567.647031:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583a000] (3)-- 00000800:00000200:13.0:1547251567.647032:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (2)-- 00000800:00000200:13.0:1547251567.647033:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583a000] (1)-- 00000800:00000200:9.0:1547251567.647046:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.647749:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251567.647754:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.647765:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251567.647766:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251567.647773:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251567.647783:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251567.648061:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251567.648069:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ae6a961000 0xa2a961000(0xa2a961000) 00000800:00000200:10.0:1547251567.648070:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba9ea4b000 0x165ea4b000(0x165ea4b000) 00000800:00000200:10.0:1547251567.648071:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba95867000 0x1655867000(0x1655867000) 00000800:00000200:10.0:1547251567.648072:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ad27969000 0x8e7969000(0x8e7969000) 00000800:00000200:10.0:1547251567.648073:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b17b02d000 0xd3b02d000(0xd3b02d000) 00000800:00000200:10.0:1547251567.648074:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b9072d7000 0x14c72d7000(0x14c72d7000) 00000800:00000200:10.0:1547251567.648075:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b17b1c2000 0xd3b1c2000(0xd3b1c2000) 00000800:00000200:10.0:1547251567.648076:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba9bea2000 0x165bea2000(0x165bea2000) 00000800:00000200:10.0:1547251567.648077:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97a9952cb000 0x5552cb000(0x5552cb000) 00000800:00000200:10.0:1547251567.648077:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b8245e9000 0x13e45e9000(0x13e45e9000) 00000800:00000200:10.0:1547251567.648078:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b82452c000 0x13e452c000(0x13e452c000) 00000800:00000200:10.0:1547251567.648079:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b4f05eb000 0x10b05eb000(0x10b05eb000) 00000800:00000200:10.0:1547251567.648080:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ae6a9fa000 0xa2a9fa000(0xa2a9fa000) 00000800:00000200:10.0:1547251567.648081:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b17b185000 0xd3b185000(0xd3b185000) 00000800:00000200:10.0:1547251567.648082:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97badf32f000 0x169f32f000(0x169f32f000) 00000800:00000200:10.0:1547251567.648083:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97badd5c0000 0x169d5c0000(0x169d5c0000) 00000800:00000200:10.0:1547251567.648083:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97a6c4ca0000 0x284ca0000(0x284ca0000) 00000800:00000200:10.0:1547251567.648084:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97aec55fb000 0xa855fb000(0xa855fb000) 00000800:00000200:9.0:1547251567.648710:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648711:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648712:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648713:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648713:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648714:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648715:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648715:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648716:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648717:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648717:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648718:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648718:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648719:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648720:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648720:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648721:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648722:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648722:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648723:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648723:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648724:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648724:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648725:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648726:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648726:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648727:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648727:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648728:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648729:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648729:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648730:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648730:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648731:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251567.648732:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251567.648733:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000100:12.0:1547251567.648982:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251567.648985:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251567.648987:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251567.649030:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:12.0:1547251567.649038:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (20)++ 00000800:00000200:12.0:1547251567.649043:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (21)-- 00000800:00000200:12.0:1547251567.649044:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:12.0:1547251567.649045:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (19)-- 00000800:00000200:12.0:1547251567.649050:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07e800] (18)-- 00000800:00000200:8.2:1547251567.649167:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e800] (17)++ 00000800:00000200:12.0:1547251567.649178:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (18)++ 00000800:00000200:12.0:1547251567.649182:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (19)-- 00000800:00000200:12.0:1547251567.649183:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (18)-- 00000800:00000200:12.0:1547251567.649184:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (17)++ 00000800:00000200:12.0:1547251567.649187:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (18)-- 00000800:00000200:12.0:1547251567.649188:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (17)-- 00000800:00000200:12.0:1547251567.649189:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (16)++ 00000800:00000200:12.0:1547251567.649191:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (17)-- 00000800:00000200:12.0:1547251567.649191:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (16)-- 00000800:00000200:13.0:1547251567.649193:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (15)++ 00000800:00000200:13.0:1547251567.649197:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (16)-- 00000800:00000200:13.0:1547251567.649198:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (15)-- 00000800:00000200:13.0:1547251567.649199:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (14)++ 00000800:00000200:13.0:1547251567.649203:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (15)-- 00000800:00000200:13.0:1547251567.649204:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (14)-- 00000800:00000200:13.0:1547251567.649205:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (13)++ 00000800:00000200:13.0:1547251567.649209:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (14)-- 00000800:00000200:13.0:1547251567.649210:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (13)-- 00000800:00000200:13.0:1547251567.649211:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (12)++ 00000800:00000200:13.0:1547251567.649214:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (13)-- 00000800:00000200:13.0:1547251567.649215:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (12)-- 00000800:00000200:13.0:1547251567.649216:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (11)++ 00000800:00000200:13.0:1547251567.649219:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (12)-- 00000800:00000200:13.0:1547251567.649220:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (11)-- 00000800:00000200:13.0:1547251567.649221:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (10)++ 00000800:00000200:13.0:1547251567.649225:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (11)-- 00000800:00000200:13.0:1547251567.649226:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (10)-- 00000800:00000200:13.0:1547251567.649227:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (9)++ 00000800:00000200:13.0:1547251567.649230:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (10)-- 00000800:00000200:13.0:1547251567.649231:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (9)-- 00000800:00000200:13.0:1547251567.649232:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (8)++ 00000800:00000200:13.0:1547251567.649235:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (9)-- 00000800:00000200:13.0:1547251567.649236:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (8)-- 00000800:00000200:13.0:1547251567.649237:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (7)++ 00000800:00000200:13.0:1547251567.649241:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (8)-- 00000800:00000200:13.0:1547251567.649242:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (7)-- 00000800:00000200:13.0:1547251567.649243:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (6)++ 00000800:00000200:13.0:1547251567.649246:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (7)-- 00000800:00000200:13.0:1547251567.649247:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (6)-- 00000800:00000200:13.0:1547251567.649248:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (5)++ 00000800:00000200:13.0:1547251567.649251:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (6)-- 00000800:00000200:13.0:1547251567.649252:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (5)-- 00000800:00000200:13.0:1547251567.649255:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (4)++ 00000800:00000200:13.0:1547251567.649258:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (5)-- 00000800:00000200:13.0:1547251567.649259:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (4)-- 00000800:00000200:13.0:1547251567.649260:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (3)++ 00000800:00000200:13.0:1547251567.649263:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (4)-- 00000800:00000200:13.0:1547251567.649264:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (3)-- 00000800:00000200:13.0:1547251567.649265:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (2)++ 00000800:00000200:13.0:1547251567.649269:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (3)-- 00000800:00000200:13.0:1547251567.649269:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (2)-- 00000800:00000200:13.0:1547251567.649271:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (1)-- 00000800:00000200:9.0:1547251567.649284:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.650103:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251567.650108:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.650116:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251567.650118:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251567.650124:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251567.650131:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251567.650374:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251567.650381:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba90f4b000 0x1650f4b000(0x1650f4b000) 00000800:00000200:10.0:1547251567.650382:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ae6a842000 0xa2a842000(0xa2a842000) 00000800:00000200:10.0:1547251567.650383:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba92fc4000 0x1652fc4000(0x1652fc4000) 00000800:00000200:10.0:1547251567.650384:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ba9ffcc000 0x165ffcc000(0x165ffcc000) 00000800:00000200:10.0:1547251567.650385:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ba9d3ea000 0x165d3ea000(0x165d3ea000) 00000800:00000200:10.0:1547251567.650386:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba9ce49000 0x165ce49000(0x165ce49000) 00000800:00000200:10.0:1547251567.650386:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97bae1cfd000 0x16a1cfd000(0x16a1cfd000) 00000800:00000200:10.0:1547251567.650387:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba90fac000 0x1650fac000(0x1650fac000) 00000800:00000200:10.0:1547251567.650389:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ac55dda000 0x815dda000(0x815dda000) 00000800:00000200:10.0:1547251567.650390:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ac55dd8000 0x815dd8000(0x815dd8000) 00000800:00000200:10.0:1547251567.650391:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ae6aaab000 0xa2aaab000(0xa2aaab000) 00000800:00000200:10.0:1547251567.650392:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ae6aaa9000 0xa2aaa9000(0xa2aaa9000) 00000800:00000200:10.0:1547251567.650393:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ae6a85a000 0xa2a85a000(0xa2a85a000) 00000800:00000200:10.0:1547251567.650394:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba95d6b000 0x1655d6b000(0x1655d6b000) 00000800:00000200:10.0:1547251567.650395:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba93832000 0x1653832000(0x1653832000) 00000800:00000200:10.0:1547251567.650396:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97bade6d7000 0x169e6d7000(0x169e6d7000) 00000800:00000200:10.0:1547251567.650397:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ad25b8c000 0x8e5b8c000(0x8e5b8c000) 00000800:00000200:10.0:1547251567.650397:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b82473c000 0x13e473c000(0x13e473c000) 00000800:00000200:9.0:1547251567.651024:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651026:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651026:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651027:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651028:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651028:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651029:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651030:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651030:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651031:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651031:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651032:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651033:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651033:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651034:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651034:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651035:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651036:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651036:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651037:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651038:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651038:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651039:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651039:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651040:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651041:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651041:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651042:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651042:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651043:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651044:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651045:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651045:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651046:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251567.651047:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251567.651047:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000100:12.0:1547251567.651244:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251567.651247:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251567.651249:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251567.651284:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:12.0:1547251567.651296:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (20)++ 00000800:00000200:12.0:1547251567.651302:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (21)-- 00000800:00000200:12.0:1547251567.651303:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:12.0:1547251567.651305:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (19)-- 00000800:00000200:12.0:1547251567.651310:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07f800] (18)-- 00000800:00000200:8.2:1547251567.651426:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f800] (17)++ 00000800:00000200:12.0:1547251567.651436:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (18)++ 00000800:00000200:12.0:1547251567.651441:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (19)-- 00000800:00000200:12.0:1547251567.651444:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (18)-- 00000800:00000200:12.0:1547251567.651446:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (17)++ 00000800:00000200:12.0:1547251567.651447:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (18)-- 00000800:00000200:12.0:1547251567.651448:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (17)-- 00000800:00000200:13.0:1547251567.651449:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (16)++ 00000800:00000200:13.0:1547251567.651455:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (17)-- 00000800:00000200:13.0:1547251567.651456:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (16)-- 00000800:00000200:13.0:1547251567.651457:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (15)++ 00000800:00000200:13.0:1547251567.651461:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (16)-- 00000800:00000200:13.0:1547251567.651462:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (15)-- 00000800:00000200:13.0:1547251567.651463:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (14)++ 00000800:00000200:13.0:1547251567.651466:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (15)-- 00000800:00000200:13.0:1547251567.651467:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (14)-- 00000800:00000200:13.0:1547251567.651468:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (13)++ 00000800:00000200:13.0:1547251567.651471:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (14)-- 00000800:00000200:13.0:1547251567.651472:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (13)-- 00000800:00000200:13.0:1547251567.651473:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (12)++ 00000800:00000200:13.0:1547251567.651477:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (13)-- 00000800:00000200:13.0:1547251567.651478:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (12)-- 00000800:00000200:13.0:1547251567.651479:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (11)++ 00000800:00000200:13.0:1547251567.651482:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (12)-- 00000800:00000200:13.0:1547251567.651483:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (11)-- 00000800:00000200:13.0:1547251567.651484:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (10)++ 00000800:00000200:13.0:1547251567.651488:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (11)-- 00000800:00000200:13.0:1547251567.651489:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (10)-- 00000800:00000200:13.0:1547251567.651490:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (9)++ 00000800:00000200:13.0:1547251567.651493:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (10)-- 00000800:00000200:13.0:1547251567.651494:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (9)-- 00000800:00000200:13.0:1547251567.651495:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (8)++ 00000800:00000200:13.0:1547251567.651499:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (9)-- 00000800:00000200:13.0:1547251567.651500:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (8)-- 00000800:00000200:13.0:1547251567.651502:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (7)++ 00000800:00000200:13.0:1547251567.651505:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (8)-- 00000800:00000200:13.0:1547251567.651506:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (7)-- 00000800:00000200:13.0:1547251567.651507:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (6)++ 00000800:00000200:13.0:1547251567.651511:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (7)-- 00000800:00000200:13.0:1547251567.651512:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (6)-- 00000800:00000200:13.0:1547251567.651513:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (5)++ 00000800:00000200:13.0:1547251567.651516:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (6)-- 00000800:00000200:13.0:1547251567.651517:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (5)-- 00000800:00000200:13.0:1547251567.651518:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (4)++ 00000800:00000200:13.0:1547251567.651521:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (5)-- 00000800:00000200:13.0:1547251567.651522:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (4)-- 00000800:00000200:13.0:1547251567.651523:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (3)++ 00000800:00000200:13.0:1547251567.651527:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (4)-- 00000800:00000200:13.0:1547251567.651527:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (3)-- 00000800:00000200:13.0:1547251567.651529:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (2)++ 00000800:00000200:13.0:1547251567.651532:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (3)-- 00000800:00000200:13.0:1547251567.651533:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (2)-- 00000800:00000200:13.0:1547251567.651534:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (1)-- 00000800:00000200:9.0:1547251567.651547:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.652393:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251567.652397:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.652406:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251567.652407:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251567.652414:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251567.652423:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251567.652668:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251567.652676:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97babd8e5000 0x167d8e5000(0x167d8e5000) 00000800:00000200:10.0:1547251567.652678:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba9298e000 0x165298e000(0x165298e000) 00000800:00000200:10.0:1547251567.652679:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97baa0f14000 0x1660f14000(0x1660f14000) 00000800:00000200:10.0:1547251567.652680:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97abf4fe9000 0x7b4fe9000(0x7b4fe9000) 00000800:00000200:10.0:1547251567.652681:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ba9a6d8000 0x165a6d8000(0x165a6d8000) 00000800:00000200:10.0:1547251567.652681:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b17b183000 0xd3b183000(0xd3b183000) 00000800:00000200:10.0:1547251567.652682:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97add79e8000 0x9979e8000(0x9979e8000) 00000800:00000200:10.0:1547251567.652683:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ae6a8c0000 0xa2a8c0000(0xa2a8c0000) 00000800:00000200:10.0:1547251567.652684:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97bae1a97000 0x16a1a97000(0x16a1a97000) 00000800:00000200:10.0:1547251567.652685:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b17b269000 0xd3b269000(0xd3b269000) 00000800:00000200:10.0:1547251567.652686:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97badf161000 0x169f161000(0x169f161000) 00000800:00000200:10.0:1547251567.652687:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba953ff000 0x16553ff000(0x16553ff000) 00000800:00000200:10.0:1547251567.652688:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba941cb000 0x16541cb000(0x16541cb000) 00000800:00000200:10.0:1547251567.652688:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b80cd13000 0x13ccd13000(0x13ccd13000) 00000800:00000200:10.0:1547251567.652689:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97a9951b2000 0x5551b2000(0x5551b2000) 00000800:00000200:10.0:1547251567.652690:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba9df06000 0x165df06000(0x165df06000) 00000800:00000200:10.0:1547251567.652691:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b17b161000 0xd3b161000(0xd3b161000) 00000800:00000200:10.0:1547251567.652692:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b824612000 0x13e4612000(0x13e4612000) 00000800:00000200:9.0:1547251567.653259:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653261:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653261:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653262:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653262:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653263:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653264:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653264:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653265:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653266:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653266:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653267:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653267:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653268:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653269:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653269:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653270:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653270:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653271:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653272:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653272:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653273:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653273:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653274:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653275:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653275:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653276:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653277:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653278:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653278:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653279:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653279:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653280:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653281:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251567.653281:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251567.653282:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000100:12.0:1547251567.653447:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251567.653450:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251567.653452:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251567.653490:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:12.0:1547251567.653502:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (20)++ 00000800:00000200:12.0:1547251567.653506:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (21)-- 00000800:00000200:12.0:1547251567.653508:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:12.0:1547251567.653509:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (19)-- 00000800:00000200:12.0:1547251567.653514:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c200] (18)-- 00000800:00000200:8.2:1547251567.653629:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c200] (17)++ 00000800:00000200:12.0:1547251567.653636:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (18)++ 00000800:00000200:12.0:1547251567.653640:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (19)-- 00000800:00000200:12.0:1547251567.653641:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (18)-- 00000800:00000200:12.0:1547251567.653642:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (17)++ 00000800:00000200:12.0:1547251567.653644:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (18)-- 00000800:00000200:12.0:1547251567.653645:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (17)-- 00000800:00000200:12.0:1547251567.653647:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (16)++ 00000800:00000200:12.0:1547251567.653648:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (17)-- 00000800:00000200:13.0:1547251567.653649:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (16)++ 00000800:00000200:12.0:1547251567.653650:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (17)-- 00000800:00000200:13.0:1547251567.653651:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (16)-- 00000800:00000200:12.0:1547251567.653651:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (16)++ 00000800:00000200:13.0:1547251567.653652:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (16)-- 00000800:00000200:13.0:1547251567.653653:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (15)++ 00000800:00000200:12.0:1547251567.653653:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (15)-- 00000800:00000200:12.0:1547251567.653654:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (15)-- 00000800:00000200:13.0:1547251567.653655:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (14)-- 00000800:00000200:12.0:1547251567.653655:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (14)++ 00000800:00000200:13.0:1547251567.653656:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (14)-- 00000800:00000200:13.0:1547251567.653657:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (13)++ 00000800:00000200:12.0:1547251567.653657:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (13)-- 00000800:00000200:12.0:1547251567.653658:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (13)-- 00000800:00000200:13.0:1547251567.653659:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (12)-- 00000800:00000200:12.0:1547251567.653660:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (12)++ 00000800:00000200:13.0:1547251567.653661:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (12)-- 00000800:00000200:12.0:1547251567.653661:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (11)-- 00000800:00000200:13.0:1547251567.653662:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (11)++ 00000800:00000200:12.0:1547251567.653663:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (11)-- 00000800:00000200:13.0:1547251567.653664:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (10)-- 00000800:00000200:12.0:1547251567.653664:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (10)++ 00000800:00000200:13.0:1547251567.653665:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (10)-- 00000800:00000200:13.0:1547251567.653666:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (9)++ 00000800:00000200:12.0:1547251567.653666:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (9)-- 00000800:00000200:12.0:1547251567.653667:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (9)-- 00000800:00000200:13.0:1547251567.653668:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (8)-- 00000800:00000200:12.0:1547251567.653668:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (8)++ 00000800:00000200:13.0:1547251567.653669:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (8)-- 00000800:00000200:13.0:1547251567.653670:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (7)++ 00000800:00000200:12.0:1547251567.653670:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (7)-- 00000800:00000200:12.0:1547251567.653671:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (7)-- 00000800:00000200:13.0:1547251567.653675:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (6)-- 00000800:00000200:12.0:1547251567.653675:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (6)++ 00000800:00000200:13.0:1547251567.653676:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (6)-- 00000800:00000200:12.0:1547251567.653677:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (5)-- 00000800:00000200:13.0:1547251567.653678:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (5)++ 00000800:00000200:12.0:1547251567.653679:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (5)-- 00000800:00000200:13.0:1547251567.653680:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (4)-- 00000800:00000200:12.0:1547251567.653680:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (4)++ 00000800:00000200:13.0:1547251567.653681:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (4)-- 00000800:00000200:13.0:1547251567.653682:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (3)-- 00000800:00000200:12.0:1547251567.653682:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (3)-- 00000800:00000200:12.0:1547251567.653684:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (1)-- 00000800:00000200:9.0:1547251567.653695:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.654301:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251567.654305:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.654313:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251567.654315:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251567.654320:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251567.654326:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251567.654565:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251567.654572:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b3b5eda000 0xf75eda000(0xf75eda000) 00000800:00000200:10.0:1547251567.654573:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba97277000 0x1657277000(0x1657277000) 00000800:00000200:10.0:1547251567.654574:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97bae1db5000 0x16a1db5000(0x16a1db5000) 00000800:00000200:10.0:1547251567.654575:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b1182b1000 0xcd82b1000(0xcd82b1000) 00000800:00000200:10.0:1547251567.654575:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ba97d7b000 0x1657d7b000(0x1657d7b000) 00000800:00000200:10.0:1547251567.654576:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba8fc43000 0x164fc43000(0x164fc43000) 00000800:00000200:10.0:1547251567.654577:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ba9ef4f000 0x165ef4f000(0x165ef4f000) 00000800:00000200:10.0:1547251567.654578:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97a99502c000 0x55502c000(0x55502c000) 00000800:00000200:10.0:1547251567.654579:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ae378f3000 0x9f78f3000(0x9f78f3000) 00000800:00000200:10.0:1547251567.654580:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97a954b5b000 0x514b5b000(0x514b5b000) 00000800:00000200:10.0:1547251567.654581:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97babd8d1000 0x167d8d1000(0x167d8d1000) 00000800:00000200:10.0:1547251567.654582:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba9df17000 0x165df17000(0x165df17000) 00000800:00000200:10.0:1547251567.654583:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b8246a0000 0x13e46a0000(0x13e46a0000) 00000800:00000200:10.0:1547251567.654583:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97af3a6e1000 0xafa6e1000(0xafa6e1000) 00000800:00000200:10.0:1547251567.654594:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97babd01d000 0x167d01d000(0x167d01d000) 00000800:00000200:10.0:1547251567.654595:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b9073ae000 0x14c73ae000(0x14c73ae000) 00000800:00000200:10.0:1547251567.654596:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97babd482000 0x167d482000(0x167d482000) 00000800:00000200:10.0:1547251567.654597:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba99f62000 0x1659f62000(0x1659f62000) 00000800:00000200:9.0:1547251567.655153:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655155:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655155:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655156:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655156:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655157:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655158:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655158:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655159:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655160:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655160:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655161:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655161:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655162:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655163:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655163:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655164:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655164:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655165:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655166:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655166:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655167:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655167:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655169:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655169:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655170:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655171:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655171:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655172:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655172:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655173:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655174:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655174:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655175:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251567.655175:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251567.655176:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000100:12.0:1547251567.655403:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251567.655406:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251567.655408:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251567.655455:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:12.0:1547251567.655465:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:13.0:1547251567.655479:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:13.0:1547251567.655486:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:13.0:1547251567.655487:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (19)-- 00000800:00000200:13.0:1547251567.655488:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (18)-- 00000800:00000200:8.2:1547251567.655597:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07ea00] (17)++ 00000800:00000200:12.0:1547251567.655605:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (18)++ 00000800:00000200:12.0:1547251567.655612:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (19)-- 00000800:00000200:12.0:1547251567.655614:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (18)-- 00000800:00000200:12.0:1547251567.655616:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (17)++ 00000800:00000200:12.0:1547251567.655617:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (18)-- 00000800:00000200:12.0:1547251567.655618:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (17)-- 00000800:00000200:12.0:1547251567.655620:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (16)++ 00000800:00000200:13.0:1547251567.655621:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (17)++ 00000800:00000200:12.0:1547251567.655621:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (17)-- 00000800:00000200:12.0:1547251567.655622:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (17)-- 00000800:00000200:13.0:1547251567.655623:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (16)-- 00000800:00000200:12.0:1547251567.655623:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (16)++ 00000800:00000200:13.0:1547251567.655624:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (16)-- 00000800:00000200:12.0:1547251567.655625:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (15)-- 00000800:00000200:13.0:1547251567.655626:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (15)++ 00000800:00000200:12.0:1547251567.655627:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (15)-- 00000800:00000200:13.0:1547251567.655628:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (14)-- 00000800:00000200:12.0:1547251567.655628:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (14)++ 00000800:00000200:13.0:1547251567.655629:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (14)-- 00000800:00000200:13.0:1547251567.655630:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (13)++ 00000800:00000200:12.0:1547251567.655630:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (13)-- 00000800:00000200:12.0:1547251567.655631:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (13)-- 00000800:00000200:13.0:1547251567.655632:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (12)-- 00000800:00000200:12.0:1547251567.655632:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (12)++ 00000800:00000200:13.0:1547251567.655633:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (12)-- 00000800:00000200:13.0:1547251567.655634:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (11)++ 00000800:00000200:12.0:1547251567.655634:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (11)-- 00000800:00000200:12.0:1547251567.655635:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (11)-- 00000800:00000200:13.0:1547251567.655636:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (10)-- 00000800:00000200:12.0:1547251567.655636:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (10)++ 00000800:00000200:13.0:1547251567.655637:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (10)-- 00000800:00000200:13.0:1547251567.655638:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (9)++ 00000800:00000200:12.0:1547251567.655638:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (9)-- 00000800:00000200:12.0:1547251567.655639:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (9)-- 00000800:00000200:13.0:1547251567.655640:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (8)-- 00000800:00000200:12.0:1547251567.655640:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (8)++ 00000800:00000200:13.0:1547251567.655641:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (8)-- 00000800:00000200:13.0:1547251567.655642:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (7)++ 00000800:00000200:12.0:1547251567.655642:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (7)-- 00000800:00000200:12.0:1547251567.655643:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (7)-- 00000800:00000200:13.0:1547251567.655644:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (6)-- 00000800:00000200:12.0:1547251567.655644:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (6)++ 00000800:00000200:13.0:1547251567.655645:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (6)-- 00000800:00000200:13.0:1547251567.655646:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (5)++ 00000800:00000200:12.0:1547251567.655646:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (5)-- 00000800:00000200:13.0:1547251567.655647:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (6)-- 00000800:00000200:13.0:1547251567.655648:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (5)-- 00000800:00000200:12.0:1547251567.655649:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (3)-- 00000800:00000200:13.0:1547251567.655650:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (2)++ 00000800:00000200:13.0:1547251567.655654:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (3)-- 00000800:00000200:13.0:1547251567.655655:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (2)-- 00000800:00000200:13.0:1547251567.655656:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (1)-- 00000800:00000200:9.0:1547251567.655664:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.656400:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251567.656404:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.656412:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251567.656413:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251567.656419:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251567.656426:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251567.656661:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251567.656669:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97badf8e3000 0x169f8e3000(0x169f8e3000) 00000800:00000200:10.0:1547251567.656670:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b3b34a7000 0xf734a7000(0xf734a7000) 00000800:00000200:10.0:1547251567.656671:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba9dfa4000 0x165dfa4000(0x165dfa4000) 00000800:00000200:10.0:1547251567.656672:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ba9f62a000 0x165f62a000(0x165f62a000) 00000800:00000200:10.0:1547251567.656673:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97bada9b9000 0x169a9b9000(0x169a9b9000) 00000800:00000200:10.0:1547251567.656673:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b17b3d5000 0xd3b3d5000(0xd3b3d5000) 00000800:00000200:10.0:1547251567.656674:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b17b3bf000 0xd3b3bf000(0xd3b3bf000) 00000800:00000200:10.0:1547251567.656676:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b1d89c5000 0xd989c5000(0xd989c5000) 00000800:00000200:10.0:1547251567.656677:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ad5720c000 0x91720c000(0x91720c000) 00000800:00000200:10.0:1547251567.656678:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba936c0000 0x16536c0000(0x16536c0000) 00000800:00000200:10.0:1547251567.656679:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97a8292e6000 0x3e92e6000(0x3e92e6000) 00000800:00000200:10.0:1547251567.656680:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ae6a91a000 0xa2a91a000(0xa2a91a000) 00000800:00000200:10.0:1547251567.656681:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba973cb000 0x16573cb000(0x16573cb000) 00000800:00000200:10.0:1547251567.656682:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b8245cc000 0x13e45cc000(0x13e45cc000) 00000800:00000200:10.0:1547251567.656683:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97a601afb000 0x1c1afb000(0x1c1afb000) 00000800:00000200:10.0:1547251567.656683:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97af80bb8000 0xb40bb8000(0xb40bb8000) 00000800:00000200:10.0:1547251567.656684:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b51bac7000 0x10dbac7000(0x10dbac7000) 00000800:00000200:10.0:1547251567.656685:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97baa2be8000 0x1662be8000(0x1662be8000) 00000800:00000200:9.0:1547251567.657254:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657256:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657256:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657257:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657257:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657258:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657259:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657259:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657260:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657261:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657261:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657262:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657262:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657263:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657264:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657264:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657265:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657266:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657266:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657267:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657268:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657268:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657269:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657270:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657270:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657271:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657271:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657272:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657273:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657273:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657274:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657274:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657275:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657276:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251567.657276:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251567.657277:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000100:12.0:1547251567.657448:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251567.657450:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251567.657452:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251567.657487:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:12.0:1547251567.657498:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (20)++ 00000800:00000200:12.0:1547251567.657503:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (21)-- 00000800:00000200:12.0:1547251567.657504:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:12.0:1547251567.657506:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (19)-- 00000800:00000200:12.0:1547251567.657511:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07f600] (18)-- 00000800:00000200:8.2:1547251567.657635:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f600] (17)++ 00000800:00000200:12.0:1547251567.657646:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (18)++ 00000800:00000200:12.0:1547251567.657650:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (19)-- 00000800:00000200:12.0:1547251567.657651:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (18)-- 00000800:00000200:12.0:1547251567.657653:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (17)++ 00000800:00000200:12.0:1547251567.657655:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (18)-- 00000800:00000200:12.0:1547251567.657656:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (17)-- 00000800:00000200:12.0:1547251567.657657:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (16)++ 00000800:00000200:12.0:1547251567.657658:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (17)-- 00000800:00000200:13.0:1547251567.657659:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (17)++ 00000800:00000200:12.0:1547251567.657660:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (17)-- 00000800:00000200:13.0:1547251567.657661:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (16)-- 00000800:00000200:12.0:1547251567.657662:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (16)++ 00000800:00000200:13.0:1547251567.657663:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (16)-- 00000800:00000200:12.0:1547251567.657663:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (15)-- 00000800:00000200:13.0:1547251567.657664:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (15)++ 00000800:00000200:12.0:1547251567.657666:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (15)-- 00000800:00000200:13.0:1547251567.657667:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (14)-- 00000800:00000200:12.0:1547251567.657667:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (14)++ 00000800:00000200:13.0:1547251567.657668:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (14)-- 00000800:00000200:12.0:1547251567.657669:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (13)-- 00000800:00000200:13.0:1547251567.657670:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (13)++ 00000800:00000200:12.0:1547251567.657671:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (13)-- 00000800:00000200:13.0:1547251567.657672:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (12)-- 00000800:00000200:12.0:1547251567.657672:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (12)++ 00000800:00000200:13.0:1547251567.657673:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (12)-- 00000800:00000200:13.0:1547251567.657674:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (11)++ 00000800:00000200:12.0:1547251567.657674:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (11)-- 00000800:00000200:12.0:1547251567.657675:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (11)-- 00000800:00000200:13.0:1547251567.657676:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (10)-- 00000800:00000200:12.0:1547251567.657676:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (10)++ 00000800:00000200:13.0:1547251567.657677:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (10)-- 00000800:00000200:13.0:1547251567.657678:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (9)++ 00000800:00000200:12.0:1547251567.657678:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (9)-- 00000800:00000200:12.0:1547251567.657679:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (9)-- 00000800:00000200:13.0:1547251567.657680:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (8)-- 00000800:00000200:12.0:1547251567.657680:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (8)++ 00000800:00000200:13.0:1547251567.657681:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (8)-- 00000800:00000200:13.0:1547251567.657682:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (7)++ 00000800:00000200:12.0:1547251567.657682:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (7)-- 00000800:00000200:12.0:1547251567.657683:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (7)-- 00000800:00000200:13.0:1547251567.657684:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (6)-- 00000800:00000200:12.0:1547251567.657684:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (6)++ 00000800:00000200:13.0:1547251567.657685:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (6)-- 00000800:00000200:13.0:1547251567.657686:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (5)++ 00000800:00000200:12.0:1547251567.657686:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (5)-- 00000800:00000200:12.0:1547251567.657687:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (5)-- 00000800:00000200:13.0:1547251567.657689:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (4)-- 00000800:00000200:12.0:1547251567.657689:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (4)++ 00000800:00000200:13.0:1547251567.657690:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (4)-- 00000800:00000200:12.0:1547251567.657691:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (3)-- 00000800:00000200:13.0:1547251567.657692:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (3)-- 00000800:00000200:12.0:1547251567.657693:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (1)-- 00000800:00000200:9.0:1547251567.657705:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.658339:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251567.658343:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.658351:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251567.658352:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251567.658358:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251567.658365:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251567.658610:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251567.658617:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97baa226d000 0x166226d000(0x166226d000) 00000800:00000200:10.0:1547251567.658618:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97bade62c000 0x169e62c000(0x169e62c000) 00000800:00000200:10.0:1547251567.658619:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba96fb3000 0x1656fb3000(0x1656fb3000) 00000800:00000200:10.0:1547251567.658620:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97bae0070000 0x16a0070000(0x16a0070000) 00000800:00000200:10.0:1547251567.658621:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ae6a9e4000 0xa2a9e4000(0xa2a9e4000) 00000800:00000200:10.0:1547251567.658622:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b6fde38000 0x12bde38000(0x12bde38000) 00000800:00000200:10.0:1547251567.658623:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b6ed5b0000 0x12ad5b0000(0x12ad5b0000) 00000800:00000200:10.0:1547251567.658623:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97badcd8d000 0x169cd8d000(0x169cd8d000) 00000800:00000200:10.0:1547251567.658624:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b645992000 0x1205992000(0x1205992000) 00000800:00000200:10.0:1547251567.658625:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba8f2da000 0x164f2da000(0x164f2da000) 00000800:00000200:10.0:1547251567.658626:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b6269f6000 0x11e69f6000(0x11e69f6000) 00000800:00000200:10.0:1547251567.658627:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b17b1b6000 0xd3b1b6000(0xd3b1b6000) 00000800:00000200:10.0:1547251567.658628:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b81d7f6000 0x13dd7f6000(0x13dd7f6000) 00000800:00000200:10.0:1547251567.658628:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b9073ac000 0x14c73ac000(0x14c73ac000) 00000800:00000200:10.0:1547251567.658629:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba50f88000 0x1610f88000(0x1610f88000) 00000800:00000200:10.0:1547251567.658630:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97badd8ae000 0x169d8ae000(0x169d8ae000) 00000800:00000200:10.0:1547251567.658631:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ba9b61f000 0x165b61f000(0x165b61f000) 00000800:00000200:10.0:1547251567.658633:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba96d43000 0x1656d43000(0x1656d43000) 00000800:00000200:9.0:1547251567.659195:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659196:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659197:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659197:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659198:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659198:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659199:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659200:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659200:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659201:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659202:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659202:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659203:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659203:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659204:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659205:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659206:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659206:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659207:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659208:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659208:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659209:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659209:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659210:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659211:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659211:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659212:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659212:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659213:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659214:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659214:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659215:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659215:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659216:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251567.659217:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251567.659217:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000100:12.0:1547251567.659400:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251567.659403:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251567.659404:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251567.659449:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:12.0:1547251567.659461:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:13.0:1547251567.659472:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:13.0:1547251567.659478:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:13.0:1547251567.659479:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (19)-- 00000800:00000200:13.0:1547251567.659481:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (18)-- 00000800:00000200:8.2:1547251567.659583:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07dc00] (17)++ 00000800:00000200:12.0:1547251567.659603:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (18)++ 00000800:00000200:12.0:1547251567.659608:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (19)-- 00000800:00000200:12.0:1547251567.659609:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (18)-- 00000800:00000200:12.0:1547251567.659611:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (17)++ 00000800:00000200:13.0:1547251567.659612:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (18)++ 00000800:00000200:12.0:1547251567.659612:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (18)-- 00000800:00000200:12.0:1547251567.659613:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (18)-- 00000800:00000200:13.0:1547251567.659615:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (17)-- 00000800:00000200:12.0:1547251567.659615:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (17)++ 00000800:00000200:13.0:1547251567.659616:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (17)-- 00000800:00000200:13.0:1547251567.659617:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (16)++ 00000800:00000200:12.0:1547251567.659617:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (16)-- 00000800:00000200:12.0:1547251567.659618:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (16)-- 00000800:00000200:13.0:1547251567.659619:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (15)-- 00000800:00000200:12.0:1547251567.659619:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (15)++ 00000800:00000200:13.0:1547251567.659620:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (15)-- 00000800:00000200:13.0:1547251567.659621:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (14)++ 00000800:00000200:12.0:1547251567.659621:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (14)-- 00000800:00000200:12.0:1547251567.659622:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (14)-- 00000800:00000200:13.0:1547251567.659623:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (13)-- 00000800:00000200:12.0:1547251567.659623:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (13)++ 00000800:00000200:13.0:1547251567.659624:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (13)-- 00000800:00000200:13.0:1547251567.659625:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (12)++ 00000800:00000200:12.0:1547251567.659625:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (12)-- 00000800:00000200:13.0:1547251567.659626:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (12)-- 00000800:00000200:12.0:1547251567.659626:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (12)-- 00000800:00000200:13.0:1547251567.659627:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (10)-- 00000800:00000200:12.0:1547251567.659628:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (9)++ 00000800:00000200:12.0:1547251567.659630:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (10)-- 00000800:00000200:12.0:1547251567.659631:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (9)-- 00000800:00000200:12.0:1547251567.659632:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (8)++ 00000800:00000200:12.0:1547251567.659635:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (9)-- 00000800:00000200:12.0:1547251567.659636:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (8)-- 00000800:00000200:12.0:1547251567.659638:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (7)++ 00000800:00000200:12.0:1547251567.659641:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (8)-- 00000800:00000200:12.0:1547251567.659642:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (7)-- 00000800:00000200:12.0:1547251567.659643:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (6)++ 00000800:00000200:12.0:1547251567.659646:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (7)-- 00000800:00000200:12.0:1547251567.659647:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (6)-- 00000800:00000200:12.0:1547251567.659648:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (5)++ 00000800:00000200:12.0:1547251567.659652:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (6)-- 00000800:00000200:13.0:1547251567.659654:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (6)++ 00000800:00000200:12.0:1547251567.659655:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (6)-- 00000800:00000200:13.0:1547251567.659656:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (5)-- 00000800:00000200:12.0:1547251567.659656:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (5)++ 00000800:00000200:13.0:1547251567.659657:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (5)-- 00000800:00000200:13.0:1547251567.659659:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (4)++ 00000800:00000200:12.0:1547251567.659659:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (4)-- 00000800:00000200:12.0:1547251567.659660:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (4)-- 00000800:00000200:13.0:1547251567.659661:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (3)-- 00000800:00000200:12.0:1547251567.659661:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (3)-- 00000800:00000200:13.0:1547251567.659663:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (1)-- 00000800:00000200:9.0:1547251567.659670:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.660558:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251567.660562:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.660571:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251567.660573:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251567.660579:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251567.660592:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251567.660829:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251567.660836:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b630fe9000 0x11f0fe9000(0x11f0fe9000) 00000800:00000200:10.0:1547251567.660838:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba936c4000 0x16536c4000(0x16536c4000) 00000800:00000200:10.0:1547251567.660838:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b81d5d0000 0x13dd5d0000(0x13dd5d0000) 00000800:00000200:10.0:1547251567.660839:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ba961c7000 0x16561c7000(0x16561c7000) 00000800:00000200:10.0:1547251567.660840:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b17b1a7000 0xd3b1a7000(0xd3b1a7000) 00000800:00000200:10.0:1547251567.660841:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba92811000 0x1652811000(0x1652811000) 00000800:00000200:10.0:1547251567.660842:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ad6bcfc000 0x92bcfc000(0x92bcfc000) 00000800:00000200:10.0:1547251567.660843:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97af82fe6000 0xb42fe6000(0xb42fe6000) 00000800:00000200:10.0:1547251567.660844:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ae6aba0000 0xa2aba0000(0xa2aba0000) 00000800:00000200:10.0:1547251567.660845:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b17b0dc000 0xd3b0dc000(0xd3b0dc000) 00000800:00000200:10.0:1547251567.660846:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b17b0df000 0xd3b0df000(0xd3b0df000) 00000800:00000200:10.0:1547251567.660846:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ae6a8c2000 0xa2a8c2000(0xa2a8c2000) 00000800:00000200:10.0:1547251567.660847:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba9beeb000 0x165beeb000(0x165beeb000) 00000800:00000200:10.0:1547251567.660848:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97badc8d5000 0x169c8d5000(0x169c8d5000) 00000800:00000200:10.0:1547251567.660849:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97bae1596000 0x16a1596000(0x16a1596000) 00000800:00000200:10.0:1547251567.660850:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97badcabe000 0x169cabe000(0x169cabe000) 00000800:00000200:10.0:1547251567.660851:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ba8cbae000 0x164cbae000(0x164cbae000) 00000800:00000200:10.0:1547251567.660851:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97baa1ad1000 0x1661ad1000(0x1661ad1000) 00000800:00000200:9.0:1547251567.661449:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661450:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661450:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661451:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661452:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661452:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661453:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661453:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661454:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661455:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661455:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661456:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661457:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661458:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661458:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661459:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661459:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661460:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661461:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661461:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661462:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661462:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661463:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661464:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661464:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661465:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661465:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661466:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661467:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661467:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661468:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661468:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661469:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661470:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251567.661470:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251567.661471:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000100:12.0:1547251567.661705:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251567.661708:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251567.661709:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251567.661748:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:12.0:1547251567.661757:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (20)++ 00000800:00000200:12.0:1547251567.661761:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (21)-- 00000800:00000200:12.0:1547251567.661762:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:12.0:1547251567.661764:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (19)-- 00000800:00000200:12.0:1547251567.661780:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07fe00] (18)-- 00000800:00000200:8.2:1547251567.661936:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07fe00] (17)++ 00000800:00000200:12.0:1547251567.661953:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (18)++ 00000800:00000200:12.0:1547251567.661957:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (19)-- 00000800:00000200:12.0:1547251567.661958:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (18)-- 00000800:00000200:12.0:1547251567.661959:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (17)++ 00000800:00000200:12.0:1547251567.661961:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (18)-- 00000800:00000200:12.0:1547251567.661962:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (17)-- 00000800:00000200:12.0:1547251567.661963:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (16)++ 00000800:00000200:12.0:1547251567.661967:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (17)-- 00000800:00000200:12.0:1547251567.661968:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (16)-- 00000800:00000200:12.0:1547251567.661969:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (15)++ 00000800:00000200:12.0:1547251567.661973:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (16)-- 00000800:00000200:12.0:1547251567.661974:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (15)-- 00000800:00000200:12.0:1547251567.661975:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (14)++ 00000800:00000200:12.0:1547251567.661978:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (15)-- 00000800:00000200:12.0:1547251567.661979:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (14)-- 00000800:00000200:12.0:1547251567.661981:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (13)++ 00000800:00000200:12.0:1547251567.661984:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (14)-- 00000800:00000200:12.0:1547251567.661985:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (13)-- 00000800:00000200:12.0:1547251567.661986:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (12)++ 00000800:00000200:12.0:1547251567.661990:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (13)-- 00000800:00000200:12.0:1547251567.661991:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (12)-- 00000800:00000200:12.0:1547251567.661992:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (11)++ 00000800:00000200:12.0:1547251567.661996:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (12)-- 00000800:00000200:12.0:1547251567.661998:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (11)-- 00000800:00000200:13.0:1547251567.661999:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (10)++ 00000800:00000200:13.0:1547251567.662004:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (11)-- 00000800:00000200:13.0:1547251567.662005:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (10)-- 00000800:00000200:13.0:1547251567.662006:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (9)++ 00000800:00000200:13.0:1547251567.662009:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (10)-- 00000800:00000200:13.0:1547251567.662010:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (9)-- 00000800:00000200:13.0:1547251567.662012:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (8)++ 00000800:00000200:13.0:1547251567.662015:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (9)-- 00000800:00000200:13.0:1547251567.662016:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (8)-- 00000800:00000200:13.0:1547251567.662017:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (7)++ 00000800:00000200:13.0:1547251567.662021:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (8)-- 00000800:00000200:13.0:1547251567.662022:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (7)-- 00000800:00000200:13.0:1547251567.662023:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (6)++ 00000800:00000200:13.0:1547251567.662026:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (7)-- 00000800:00000200:13.0:1547251567.662027:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (6)-- 00000800:00000200:13.0:1547251567.662028:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (5)++ 00000800:00000200:13.0:1547251567.662031:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (6)-- 00000800:00000200:13.0:1547251567.662032:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (5)-- 00000800:00000200:13.0:1547251567.662033:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (4)++ 00000800:00000200:13.0:1547251567.662037:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (5)-- 00000800:00000200:13.0:1547251567.662037:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (4)-- 00000800:00000200:13.0:1547251567.662039:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (3)++ 00000800:00000200:13.0:1547251567.662042:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (4)-- 00000800:00000200:13.0:1547251567.662043:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (3)-- 00000800:00000200:13.0:1547251567.662044:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (2)++ 00000800:00000200:13.0:1547251567.662047:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (3)-- 00000800:00000200:13.0:1547251567.662048:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (2)-- 00000800:00000200:13.0:1547251567.662050:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (1)-- 00000800:00000200:9.0:1547251567.662063:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.662817:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251567.662822:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.662831:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251567.662832:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:14.0:1547251567.662839:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:14.0:1547251567.662848:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:14.0:1547251567.663056:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:14.0:1547251567.663065:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba9c79b000 0x165c79b000(0x165c79b000) 00000800:00000200:14.0:1547251567.663066:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97af3a6e1000 0xafa6e1000(0xafa6e1000) 00000800:00000200:14.0:1547251567.663068:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97babd01d000 0x167d01d000(0x167d01d000) 00000800:00000200:14.0:1547251567.663069:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b9073ae000 0x14c73ae000(0x14c73ae000) 00000800:00000200:14.0:1547251567.663070:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97babd482000 0x167d482000(0x167d482000) 00000800:00000200:14.0:1547251567.663071:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba99f62000 0x1659f62000(0x1659f62000) 00000800:00000200:14.0:1547251567.663072:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97babd8e5000 0x167d8e5000(0x167d8e5000) 00000800:00000200:14.0:1547251567.663072:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba9298e000 0x165298e000(0x165298e000) 00000800:00000200:14.0:1547251567.663073:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97baa0f14000 0x1660f14000(0x1660f14000) 00000800:00000200:14.0:1547251567.663074:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97abf4fe9000 0x7b4fe9000(0x7b4fe9000) 00000800:00000200:14.0:1547251567.663075:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ba9a6d8000 0x165a6d8000(0x165a6d8000) 00000800:00000200:14.0:1547251567.663076:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ae6a8c0000 0xa2a8c0000(0xa2a8c0000) 00000800:00000200:14.0:1547251567.663077:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97bae1a97000 0x16a1a97000(0x16a1a97000) 00000800:00000200:14.0:1547251567.663078:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b17b269000 0xd3b269000(0xd3b269000) 00000800:00000200:14.0:1547251567.663079:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97badf161000 0x169f161000(0x169f161000) 00000800:00000200:14.0:1547251567.663080:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba953ff000 0x16553ff000(0x16553ff000) 00000800:00000200:14.0:1547251567.663080:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ba941cb000 0x16541cb000(0x16541cb000) 00000800:00000200:14.0:1547251567.663081:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba9df06000 0x165df06000(0x165df06000) 00000800:00000200:9.0:1547251567.663707:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663708:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663709:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663710:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663710:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663711:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663712:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663713:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663713:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663714:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663715:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663715:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663716:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663716:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663717:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663718:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663718:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663719:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663719:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663720:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663721:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663721:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663722:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663723:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663723:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663724:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663724:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663725:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663726:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663726:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663727:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663727:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663728:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663729:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251567.663729:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251567.663730:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000100:12.0:1547251567.663960:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251567.663963:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251567.663964:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251567.664002:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:12.0:1547251567.664009:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (20)++ 00000800:00000200:12.0:1547251567.664014:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (21)-- 00000800:00000200:12.0:1547251567.664015:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:12.0:1547251567.664017:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (19)-- 00000800:00000200:12.0:1547251567.664022:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97babe8ac600] (18)-- 00000800:00000200:8.2:1547251567.664144:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ac600] (17)++ 00000800:00000200:12.0:1547251567.664155:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (18)++ 00000800:00000200:12.0:1547251567.664159:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (19)-- 00000800:00000200:12.0:1547251567.664160:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (18)-- 00000800:00000200:12.0:1547251567.664162:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (17)++ 00000800:00000200:12.0:1547251567.664164:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (18)-- 00000800:00000200:12.0:1547251567.664165:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (17)-- 00000800:00000200:12.0:1547251567.664166:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (16)++ 00000800:00000200:12.0:1547251567.664168:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (17)-- 00000800:00000200:12.0:1547251567.664169:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (16)-- 00000800:00000200:13.0:1547251567.664170:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (15)++ 00000800:00000200:13.0:1547251567.664175:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (16)-- 00000800:00000200:13.0:1547251567.664176:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (15)-- 00000800:00000200:13.0:1547251567.664177:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (14)++ 00000800:00000200:13.0:1547251567.664180:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (15)-- 00000800:00000200:13.0:1547251567.664181:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (14)-- 00000800:00000200:13.0:1547251567.664184:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (13)++ 00000800:00000200:13.0:1547251567.664187:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (14)-- 00000800:00000200:13.0:1547251567.664188:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (13)-- 00000800:00000200:13.0:1547251567.664189:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (12)++ 00000800:00000200:13.0:1547251567.664193:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (13)-- 00000800:00000200:13.0:1547251567.664194:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (12)-- 00000800:00000200:13.0:1547251567.664195:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (11)++ 00000800:00000200:13.0:1547251567.664198:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (12)-- 00000800:00000200:13.0:1547251567.664199:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (11)-- 00000800:00000200:13.0:1547251567.664200:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (10)++ 00000800:00000200:13.0:1547251567.664204:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (11)-- 00000800:00000200:13.0:1547251567.664205:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (10)-- 00000800:00000200:13.0:1547251567.664206:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (9)++ 00000800:00000200:13.0:1547251567.664209:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (10)-- 00000800:00000200:13.0:1547251567.664210:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (9)-- 00000800:00000200:13.0:1547251567.664211:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (8)++ 00000800:00000200:13.0:1547251567.664214:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (9)-- 00000800:00000200:13.0:1547251567.664215:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (8)-- 00000800:00000200:13.0:1547251567.664216:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (7)++ 00000800:00000200:13.0:1547251567.664220:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (8)-- 00000800:00000200:13.0:1547251567.664221:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (7)-- 00000800:00000200:13.0:1547251567.664222:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (6)++ 00000800:00000200:13.0:1547251567.664225:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (7)-- 00000800:00000200:13.0:1547251567.664226:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (6)-- 00000800:00000200:13.0:1547251567.664227:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (5)++ 00000800:00000200:13.0:1547251567.664230:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (6)-- 00000800:00000200:13.0:1547251567.664231:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (5)-- 00000800:00000200:13.0:1547251567.664233:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (4)++ 00000800:00000200:13.0:1547251567.664236:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (5)-- 00000800:00000200:13.0:1547251567.664237:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (4)-- 00000800:00000200:13.0:1547251567.664238:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (3)++ 00000800:00000200:13.0:1547251567.664241:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (4)-- 00000800:00000200:13.0:1547251567.664242:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (3)-- 00000800:00000200:13.0:1547251567.664243:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (2)++ 00000800:00000200:13.0:1547251567.664246:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (3)-- 00000800:00000200:13.0:1547251567.664247:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (2)-- 00000800:00000200:13.0:1547251567.664249:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (1)-- 00000800:00000200:9.0:1547251567.664262:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.665073:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251567.665077:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.665085:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251567.665086:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251567.665093:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251567.665102:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251567.665275:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251567.665285:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba9404f000 0x165404f000(0x165404f000) 00000800:00000200:10.0:1547251567.665287:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97a6019fb000 0x1c19fb000(0x1c19fb000) 00000800:00000200:10.0:1547251567.665288:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b17b015000 0xd3b015000(0xd3b015000) 00000800:00000200:10.0:1547251567.665289:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b356e74000 0xf16e74000(0xf16e74000) 00000800:00000200:10.0:1547251567.665290:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ae6a99c000 0xa2a99c000(0xa2a99c000) 00000800:00000200:10.0:1547251567.665290:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b824557000 0x13e4557000(0x13e4557000) 00000800:00000200:10.0:1547251567.665292:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97baa0725000 0x1660725000(0x1660725000) 00000800:00000200:10.0:1547251567.665293:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba90b3d000 0x1650b3d000(0x1650b3d000) 00000800:00000200:10.0:1547251567.665294:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b6d046a000 0x129046a000(0x129046a000) 00000800:00000200:10.0:1547251567.665295:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97badf8e3000 0x169f8e3000(0x169f8e3000) 00000800:00000200:10.0:1547251567.665295:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b3b34a7000 0xf734a7000(0xf734a7000) 00000800:00000200:10.0:1547251567.665296:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba9dfa4000 0x165dfa4000(0x165dfa4000) 00000800:00000200:10.0:1547251567.665297:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba9f62a000 0x165f62a000(0x165f62a000) 00000800:00000200:10.0:1547251567.665298:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97bada9b9000 0x169a9b9000(0x169a9b9000) 00000800:00000200:10.0:1547251567.665299:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b17b3d5000 0xd3b3d5000(0xd3b3d5000) 00000800:00000200:10.0:1547251567.665300:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b1d89c5000 0xd989c5000(0xd989c5000) 00000800:00000200:10.0:1547251567.665300:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ad5720c000 0x91720c000(0x91720c000) 00000800:00000200:10.0:1547251567.665301:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba936c0000 0x16536c0000(0x16536c0000) 00000800:00000200:9.0:1547251567.665850:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665851:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665852:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665853:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665854:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665854:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665855:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665856:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665856:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665857:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665858:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665858:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665859:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665859:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665860:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665861:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665861:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665862:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665863:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665863:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665864:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665864:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665865:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665866:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665866:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665867:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665867:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665868:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665868:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665869:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665870:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665870:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665871:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665872:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251567.665872:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251567.665873:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000100:12.0:1547251567.666108:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251567.666111:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251567.666112:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251567.666162:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:12.0:1547251567.666167:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:12.0:1547251567.666171:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:12.0:1547251567.666176:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:12.0:1547251567.666177:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (19)-- 00000800:00000200:12.0:1547251567.666178:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:8.2:1547251567.666302:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c600] (17)++ 00000800:00000200:12.0:1547251567.666313:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)++ 00000800:00000200:12.0:1547251567.666317:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (19)-- 00000800:00000200:12.0:1547251567.666318:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:12.0:1547251567.666319:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (17)++ 00000800:00000200:12.0:1547251567.666321:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:12.0:1547251567.666322:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (17)-- 00000800:00000200:12.0:1547251567.666323:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (16)++ 00000800:00000200:12.0:1547251567.666327:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (17)-- 00000800:00000200:12.0:1547251567.666328:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (16)-- 00000800:00000200:12.0:1547251567.666330:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)++ 00000800:00000200:12.0:1547251567.666334:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (16)-- 00000800:00000200:12.0:1547251567.666335:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)-- 00000800:00000200:12.0:1547251567.666336:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (14)++ 00000800:00000200:12.0:1547251567.666340:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (15)-- 00000800:00000200:12.0:1547251567.666341:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (14)-- 00000800:00000200:12.0:1547251567.666342:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)++ 00000800:00000200:12.0:1547251567.666345:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (14)-- 00000800:00000200:12.0:1547251567.666346:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:12.0:1547251567.666348:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)++ 00000800:00000200:12.0:1547251567.666351:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:12.0:1547251567.666352:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)-- 00000800:00000200:12.0:1547251567.666353:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)++ 00000800:00000200:12.0:1547251567.666357:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (12)-- 00000800:00000200:12.0:1547251567.666358:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)-- 00000800:00000200:12.0:1547251567.666359:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (10)++ 00000800:00000200:12.0:1547251567.666362:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (11)-- 00000800:00000200:12.0:1547251567.666363:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (10)-- 00000800:00000200:12.0:1547251567.666364:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (9)++ 00000800:00000200:12.0:1547251567.666368:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (10)-- 00000800:00000200:12.0:1547251567.666369:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (9)-- 00000800:00000200:12.0:1547251567.666370:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)++ 00000800:00000200:12.0:1547251567.666373:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (9)-- 00000800:00000200:12.0:1547251567.666374:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:12.0:1547251567.666375:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (7)++ 00000800:00000200:12.0:1547251567.666376:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:13.0:1547251567.666377:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)++ 00000800:00000200:12.0:1547251567.666378:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:13.0:1547251567.666379:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (7)-- 00000800:00000200:12.0:1547251567.666379:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (7)++ 00000800:00000200:13.0:1547251567.666380:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (7)-- 00000800:00000200:12.0:1547251567.666381:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (6)-- 00000800:00000200:13.0:1547251567.666382:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (6)++ 00000800:00000200:13.0:1547251567.666383:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (5)-- 00000800:00000200:12.0:1547251567.666383:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (6)-- 00000800:00000200:12.0:1547251567.666384:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)++ 00000800:00000200:13.0:1547251567.666385:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)-- 00000800:00000200:13.0:1547251567.666386:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (4)++ 00000800:00000200:12.0:1547251567.666386:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (4)-- 00000800:00000200:12.0:1547251567.666387:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (4)-- 00000800:00000200:13.0:1547251567.666388:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (3)-- 00000800:00000200:12.0:1547251567.666388:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (3)-- 00000800:00000200:13.0:1547251567.666389:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (1)-- 00000800:00000200:9.0:1547251567.666401:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251567.667152:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000100:00000200:9.0:1547251567.978685:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443248, portal 4 00000100:00000200:9.0:1547251567.978688:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443248, offset 0 00000400:00000200:9.0:1547251567.978692:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251567.978694:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251567.978697:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03e70 00000400:00000200:9.0:1547251567.978699:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd40f9 [1] + 0 00000400:00000200:9.0:1547251567.978701:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ae6a914330 00000100:00000200:9.0:1547251567.978703:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251567.978712:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914330 00000400:00000200:9.0:1547251567.978713:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251567.978717:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf1500 x1622384012443248/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251567.978723:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea7f8 00000400:00000200:9.0:1547251567.978724:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251567.978726:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251567.978734:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443248 00000100:00000200:6.0:1547251567.978738:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443264, portal 4 00000100:00000200:6.0:1547251567.978741:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443264, offset 0 00010000:00000200:8.0:1547251567.978744:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3dd850 x1622384012443248/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:628/0 lens 224/368 e 0 to 0 dl 1547251573 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:6.0:1547251567.978744:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251567.978747:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251567.978749:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03e80 00000100:00000200:8.0:1547251567.978751:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443248, offset 192 00000400:00000200:8.0:1547251567.978753:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251567.978753:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1cf5 [1] + 0 00000400:00000200:6.0:1547251567.978755:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad660 00000400:00000200:8.0:1547251567.978756:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:6.0:1547251567.978757:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251567.978759:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03e70 00000400:00000200:8.0:1547251567.978761:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaea51 [1] + 192 00000100:00000200:8.0:1547251567.978764:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf1500 x1622384012443248/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251567.978766:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad660 00000400:00000200:6.0:1547251567.978767:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251567.978770:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37930f00 x1622384012443264/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251567.978772:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251567.978774:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459906e8 00000400:00000200:8.0:1547251567.978775:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688088 00000400:00000200:6.0:1547251567.978775:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251567.978776:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:8.0:1547251567.978777:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251567.978779:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251567.978782:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf1500 x1622384012443248/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251567.978786:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:9.0:1547251567.978787:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf1500 x1622384012443248/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251567.978789:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443264 00000400:00000200:9.0:1547251567.978792:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea2a8 00010000:00080000:9.0:1547251567.978796:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf1500 x1622384012443248/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/0 rc 0/0 00010000:00000200:4.0:1547251567.978802:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c118c50 x1622384012443264/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:628/0 lens 224/368 e 0 to 0 dl 1547251573 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251567.978811:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443264, offset 192 00000400:00000200:4.0:1547251567.978814:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251567.978818:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251567.978821:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03e80 00000400:00000200:4.0:1547251567.978824:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaea61 [1] + 192 00000100:00000200:4.0:1547251567.978827:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37930f00 x1622384012443264/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251567.978837:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251567.978839:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b880 00000400:00000200:4.0:1547251567.978840:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251567.978841:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251567.978846:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37930f00 x1622384012443264/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251567.978847:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251567.978850:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37930f00 x1622384012443264/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251567.978855:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459904c8 00010000:00080000:6.0:1547251567.978858:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37930f00 x1622384012443264/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251567.994674:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443280, portal 4 00000100:00000200:6.0:1547251567.994677:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443280, offset 0 00000400:00000200:6.0:1547251567.994679:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251567.994682:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251567.994685:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03e90 00000400:00000200:6.0:1547251567.994687:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1d25 [1] + 0 00000400:00000200:6.0:1547251567.994689:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad3b8 00000100:00000200:6.0:1547251567.994691:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251567.994700:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad3b8 00000400:00000200:6.0:1547251567.994701:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251567.994703:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37937200 x1622384012443280/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251567.994708:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459906e8 00000400:00000200:6.0:1547251567.994709:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251567.994710:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251567.994733:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443280 00010000:00000200:5.0:1547251567.994742:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c118850 x1622384012443280/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:628/0 lens 224/368 e 0 to 0 dl 1547251573 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251567.994749:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443280, offset 192 00000400:00000200:5.0:1547251567.994751:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251567.994754:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251567.994757:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03e90 00000400:00000200:5.0:1547251567.994760:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaea71 [1] + 192 00000100:00000200:5.0:1547251567.994763:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37937200 x1622384012443280/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251567.994777:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251567.994779:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad7f8 00000400:00000200:5.0:1547251567.994780:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251567.994781:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251567.994784:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37937200 x1622384012443280/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251567.994786:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251567.994791:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37937200 x1622384012443280/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251567.994794:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459904c8 00010000:00080000:6.0:1547251567.994798:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37937200 x1622384012443280/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251574 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:9.0:1547251568.307673:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251568.307687:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251568.307688:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:14.0:1547251568.307696:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:14.0:1547251568.307706:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:14.0:1547251568.308234:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:14.0:1547251568.308244:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba9ffcc000 0x165ffcc000(0x165ffcc000) 00000800:00000200:14.0:1547251568.308246:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba9d3ea000 0x165d3ea000(0x165d3ea000) 00000800:00000200:14.0:1547251568.308247:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba9ce49000 0x165ce49000(0x165ce49000) 00000800:00000200:14.0:1547251568.308248:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97bae1cfd000 0x16a1cfd000(0x16a1cfd000) 00000800:00000200:14.0:1547251568.308249:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ba90fac000 0x1650fac000(0x1650fac000) 00000800:00000200:14.0:1547251568.308249:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ac55dda000 0x815dda000(0x815dda000) 00000800:00000200:14.0:1547251568.308250:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ac55dd8000 0x815dd8000(0x815dd8000) 00000800:00000200:14.0:1547251568.308251:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ae6aaa9000 0xa2aaa9000(0xa2aaa9000) 00000800:00000200:14.0:1547251568.308252:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ba95d6b000 0x1655d6b000(0x1655d6b000) 00000800:00000200:14.0:1547251568.308253:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba93832000 0x1653832000(0x1653832000) 00000800:00000200:14.0:1547251568.308254:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97bade6d7000 0x169e6d7000(0x169e6d7000) 00000800:00000200:14.0:1547251568.308255:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b82473c000 0x13e473c000(0x13e473c000) 00000800:00000200:14.0:1547251568.308256:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b9072d7000 0x14c72d7000(0x14c72d7000) 00000800:00000200:14.0:1547251568.308257:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba9bea2000 0x165bea2000(0x165bea2000) 00000800:00000200:14.0:1547251568.308258:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97a9952cb000 0x5552cb000(0x5552cb000) 00000800:00000200:14.0:1547251568.308259:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97badf32f000 0x169f32f000(0x169f32f000) 00000800:00000200:14.0:1547251568.308259:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97badd5c0000 0x169d5c0000(0x169d5c0000) 00000800:00000200:14.0:1547251568.308260:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97a6c4ca0000 0x284ca0000(0x284ca0000) 00000800:00000200:9.0:1547251568.308976:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308978:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308979:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308979:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308980:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308980:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308981:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308982:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308982:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308983:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308983:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308985:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308985:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308986:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308986:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308987:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308988:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308988:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308989:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308990:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308990:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308991:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308991:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308992:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308993:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308993:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308994:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308994:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308995:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308996:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308996:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308997:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308997:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308998:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:9.0:1547251568.308999:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:9.0:1547251568.308999:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000100:12.0:1547251568.309312:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251568.309315:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251568.309318:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251568.309372:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:12.0:1547251568.309379:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:12.0:1547251568.309383:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (19)++ 00000800:00000200:12.0:1547251568.309389:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (20)-- 00000800:00000200:12.0:1547251568.309390:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (19)-- 00000800:00000200:12.0:1547251568.309392:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (18)-- 00000800:00000200:8.2:1547251568.309494:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ac000] (17)++ 00000800:00000200:12.0:1547251568.309505:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (18)++ 00000800:00000200:12.0:1547251568.309510:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (19)-- 00000800:00000200:12.0:1547251568.309511:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (18)-- 00000800:00000200:12.0:1547251568.309512:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (17)++ 00000800:00000200:12.0:1547251568.309514:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (18)-- 00000800:00000200:12.0:1547251568.309515:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (17)-- 00000800:00000200:12.0:1547251568.309517:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (16)++ 00000800:00000200:12.0:1547251568.309518:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (17)-- 00000800:00000200:12.0:1547251568.309519:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (16)-- 00000800:00000200:13.0:1547251568.309521:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (15)++ 00000800:00000200:13.0:1547251568.309526:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (16)-- 00000800:00000200:13.0:1547251568.309527:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (15)-- 00000800:00000200:13.0:1547251568.309528:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (14)++ 00000800:00000200:13.0:1547251568.309532:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (15)-- 00000800:00000200:13.0:1547251568.309533:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (14)-- 00000800:00000200:13.0:1547251568.309534:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (13)++ 00000800:00000200:13.0:1547251568.309537:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (14)-- 00000800:00000200:13.0:1547251568.309539:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (13)-- 00000800:00000200:13.0:1547251568.309540:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (12)++ 00000800:00000200:13.0:1547251568.309543:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (13)-- 00000800:00000200:13.0:1547251568.309544:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (12)-- 00000800:00000200:13.0:1547251568.309545:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (11)++ 00000800:00000200:13.0:1547251568.309548:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (12)-- 00000800:00000200:13.0:1547251568.309549:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (11)-- 00000800:00000200:13.0:1547251568.309550:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (10)++ 00000800:00000200:13.0:1547251568.309554:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (11)-- 00000800:00000200:13.0:1547251568.309555:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (10)-- 00000800:00000200:13.0:1547251568.309556:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (9)++ 00000800:00000200:13.0:1547251568.309559:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (10)-- 00000800:00000200:13.0:1547251568.309560:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (9)-- 00000800:00000200:13.0:1547251568.309561:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (8)++ 00000800:00000200:13.0:1547251568.309565:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (9)-- 00000800:00000200:13.0:1547251568.309566:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (8)-- 00000800:00000200:13.0:1547251568.309567:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (7)++ 00000800:00000200:13.0:1547251568.309570:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (8)-- 00000800:00000200:13.0:1547251568.309571:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (7)-- 00000800:00000200:13.0:1547251568.309572:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (6)++ 00000800:00000200:13.0:1547251568.309577:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (7)-- 00000800:00000200:13.0:1547251568.309578:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (6)-- 00000800:00000200:13.0:1547251568.309579:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (5)++ 00000800:00000200:12.0:1547251568.309593:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (6)++ 00000800:00000200:12.0:1547251568.309594:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (7)-- 00000800:00000200:12.0:1547251568.309595:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (6)-- 00000800:00000200:12.0:1547251568.309596:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (5)++ 00000800:00000200:12.0:1547251568.309597:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (6)-- 00000800:00000200:13.0:1547251568.309598:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (6)-- 00000800:00000200:12.0:1547251568.309598:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (5)-- 00000800:00000200:13.0:1547251568.309599:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (3)-- 00000800:00000200:12.0:1547251568.309600:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac000] (2)++ 00000800:00000200:12.0:1547251568.309604:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac000] (3)-- 00000800:00000200:12.0:1547251568.309605:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (2)-- 00000800:00000200:12.0:1547251568.309606:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac000] (1)-- 00000800:00000200:9.0:1547251568.309621:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251568.310363:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251569.307664:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251569.307677:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251569.307679:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:14.0:1547251569.307687:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:14.0:1547251569.307696:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:14.0:1547251569.308257:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:14.0:1547251569.308265:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba99e40000 0x1659e40000(0x1659e40000) 00000800:00000200:14.0:1547251569.308266:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b4f05e0000 0x10b05e0000(0x10b05e0000) 00000800:00000200:14.0:1547251569.308267:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b907301000 0x14c7301000(0x14c7301000) 00000800:00000200:14.0:1547251569.308268:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97badfab2000 0x169fab2000(0x169fab2000) 00000800:00000200:14.0:1547251569.308269:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97badcc4a000 0x169cc4a000(0x169cc4a000) 00000800:00000200:14.0:1547251569.308270:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97bade651000 0x169e651000(0x169e651000) 00000800:00000200:14.0:1547251569.308271:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ba96fb2000 0x1656fb2000(0x1656fb2000) 00000800:00000200:14.0:1547251569.308272:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba95254000 0x1655254000(0x1655254000) 00000800:00000200:14.0:1547251569.308273:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b17b2e0000 0xd3b2e0000(0xd3b2e0000) 00000800:00000200:14.0:1547251569.308273:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba9f7a5000 0x165f7a5000(0x165f7a5000) 00000800:00000200:14.0:1547251569.308274:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ad25ba3000 0x8e5ba3000(0x8e5ba3000) 00000800:00000200:14.0:1547251569.308275:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ae6a9e6000 0xa2a9e6000(0xa2a9e6000) 00000800:00000200:14.0:1547251569.308276:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba8f33a000 0x164f33a000(0x164f33a000) 00000800:00000200:14.0:1547251569.308277:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97baa034f000 0x166034f000(0x166034f000) 00000800:00000200:14.0:1547251569.308278:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba8fced000 0x164fced000(0x164fced000) 00000800:00000200:14.0:1547251569.308279:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97baa2050000 0x1662050000(0x1662050000) 00000800:00000200:14.0:1547251569.308279:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97bad9437000 0x1699437000(0x1699437000) 00000800:00000200:14.0:1547251569.308280:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ae6a81c000 0xa2a81c000(0xa2a81c000) 00000800:00000200:9.0:1547251569.308953:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308955:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308955:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308956:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308957:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308957:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308958:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308959:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308960:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308960:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308961:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308962:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308962:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308963:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308963:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308964:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308965:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308965:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308966:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308967:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308967:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308968:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308968:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308969:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308970:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308970:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308971:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308971:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308972:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308973:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308973:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308974:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308974:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308975:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251569.308975:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251569.308976:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000100:12.0:1547251569.309265:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251569.309268:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251569.309270:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251569.309324:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:12.0:1547251569.309346:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:13.0:1547251569.309350:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:13.0:1547251569.309357:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:13.0:1547251569.309358:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (19)-- 00000800:00000200:13.0:1547251569.309359:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (18)-- 00000800:00000200:8.2:1547251569.309476:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ace00] (17)++ 00000800:00000200:12.0:1547251569.309487:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (18)++ 00000800:00000200:12.0:1547251569.309492:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (19)-- 00000800:00000200:12.0:1547251569.309493:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (18)-- 00000800:00000200:12.0:1547251569.309496:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (17)++ 00000800:00000200:12.0:1547251569.309497:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (18)-- 00000800:00000200:12.0:1547251569.309498:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (17)-- 00000800:00000200:12.0:1547251569.309500:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (16)++ 00000800:00000200:12.0:1547251569.309502:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (17)-- 00000800:00000200:13.0:1547251569.309503:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (17)++ 00000800:00000200:12.0:1547251569.309504:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (17)-- 00000800:00000200:13.0:1547251569.309505:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (16)-- 00000800:00000200:12.0:1547251569.309505:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (16)++ 00000800:00000200:13.0:1547251569.309506:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (16)-- 00000800:00000200:13.0:1547251569.309507:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (15)++ 00000800:00000200:12.0:1547251569.309507:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (15)-- 00000800:00000200:12.0:1547251569.309508:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (15)-- 00000800:00000200:13.0:1547251569.309509:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (14)-- 00000800:00000200:12.0:1547251569.309509:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (14)++ 00000800:00000200:13.0:1547251569.309510:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (14)-- 00000800:00000200:13.0:1547251569.309511:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (13)++ 00000800:00000200:12.0:1547251569.309511:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (13)-- 00000800:00000200:12.0:1547251569.309512:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (13)-- 00000800:00000200:13.0:1547251569.309513:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (12)-- 00000800:00000200:13.0:1547251569.309514:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (12)-- 00000800:00000200:12.0:1547251569.309514:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (12)++ 00000800:00000200:12.0:1547251569.309515:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (11)-- 00000800:00000200:13.0:1547251569.309516:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (11)++ 00000800:00000200:13.0:1547251569.309517:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (10)-- 00000800:00000200:12.0:1547251569.309517:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (11)-- 00000800:00000200:12.0:1547251569.309518:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (10)++ 00000800:00000200:13.0:1547251569.309519:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (10)-- 00000800:00000200:12.0:1547251569.309519:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (9)-- 00000800:00000200:13.0:1547251569.309520:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (9)++ 00000800:00000200:13.0:1547251569.309521:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (8)-- 00000800:00000200:12.0:1547251569.309521:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (9)-- 00000800:00000200:12.0:1547251569.309522:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (8)++ 00000800:00000200:13.0:1547251569.309523:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (8)-- 00000800:00000200:12.0:1547251569.309523:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (7)-- 00000800:00000200:13.0:1547251569.309524:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (7)++ 00000800:00000200:12.0:1547251569.309524:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (7)-- 00000800:00000200:13.0:1547251569.309525:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (6)-- 00000800:00000200:13.0:1547251569.309526:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (6)-- 00000800:00000200:12.0:1547251569.309526:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (6)++ 00000800:00000200:12.0:1547251569.309527:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (5)-- 00000800:00000200:13.0:1547251569.309528:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (5)++ 00000800:00000200:12.0:1547251569.309528:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (5)-- 00000800:00000200:13.0:1547251569.309529:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (4)-- 00000800:00000200:13.0:1547251569.309530:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (4)-- 00000800:00000200:12.0:1547251569.309530:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (4)++ 00000800:00000200:12.0:1547251569.309531:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (3)-- 00000800:00000200:13.0:1547251569.309532:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (3)-- 00000800:00000200:12.0:1547251569.309533:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (1)-- 00000800:00000200:9.0:1547251569.309546:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251569.310156:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000100:00000200:8.0:1547251570.010676:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443296, portal 4 00000100:00000200:8.0:1547251570.010679:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443296, offset 0 00000400:00000200:8.0:1547251570.010682:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000100:00000200:15.0:1547251570.010685:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443312, portal 4 00000400:00000200:8.0:1547251570.010685:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:8.0:1547251570.010687:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22997] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:15.0:1547251570.010688:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443312, offset 0 00000400:00000200:8.0:1547251570.010690:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22995] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22997] 00000400:00000200:15.0:1547251570.010691:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:8.0:1547251570.010693:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:15.0:1547251570.010694:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:8.0:1547251570.010695:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:1.0:1547251570.010695:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443328, portal 4 00000400:00000200:8.0:1547251570.010696:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251570.010697:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22998] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251570.010697:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9728, 9727] 00000100:00000200:1.0:1547251570.010698:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443328, offset 0 00000400:00000200:8.0:1547251570.010699:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:15.0:1547251570.010700:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22995] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22998] 00000400:00000200:1.0:1547251570.010701:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:15.0:1547251570.010702:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:15.0:1547251570.010704:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251570.010704:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:15.0:1547251570.010707:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251570.010707:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:22999] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251570.010708:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9728, 9728] 00000400:00000200:15.0:1547251570.010710:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:1.0:1547251570.010710:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22995] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:22999] 00000400:00000200:1.0:1547251570.010712:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:1.0:1547251570.010714:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251570.010715:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000400:00000200:1.0:1547251570.010716:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251570.010717:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9728, 9729] 00000800:00000200:15.0:1547251570.010718:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:1.0:1547251570.010719:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:8.0:1547251570.010722:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251570.010724:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:15.0:1547251570.010726:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251570.010728:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:8.0:1547251570.010728:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251570.010729:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000400:00000200:1.0:1547251570.010729:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251570.010730:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (21)++ 00000800:00000200:15.0:1547251570.010732:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (23)-- 00000800:00000200:8.0:1547251570.010732:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (23)++ 00000800:00000200:1.0:1547251570.010732:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:8.0:1547251570.010735:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (23)-- 00000800:00000200:1.0:1547251570.010735:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251570.010736:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:1.0:1547251570.010737:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:1.0:1547251570.010740:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:7.2:1547251570.010784:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (22)++ 00000800:00000200:7.2:1547251570.010793:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251570.010808:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (23)++ 00000800:00000200:13.0:1547251570.010811:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251570.010812:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (24)-- 00000800:00000200:13.0:1547251570.010813:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:13.0:1547251570.010817:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b5700 x1622384012443328/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251577 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251570.010817:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cae000 x1622384012443312/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251577 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251570.010824:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2000 00000400:00000200:12.0:1547251570.010824:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1770 00000400:00000200:13.0:1547251570.010826:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251570.010826:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251570.010829:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:13.0:1547251570.010831:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251570.010832:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (23)-- 00000800:00000200:13.0:1547251570.010834:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251570.010834:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:13.0:1547251570.010835:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:12.0:1547251570.010836:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000800:00000200:13.0:1547251570.010837:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000100:00000200:12.0:1547251570.010838:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824756f00 x1622384012443296/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251577 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251570.010850:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000400:00000200:12.0:1547251570.010851:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251570.010854:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251570.010855:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:7.2:1547251570.010906:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251570.010911:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:13.0:1547251570.010919:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:13.0:1547251570.010923:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251570.010925:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:13.0:1547251570.010928:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251570.010928:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000800:00000200:9.2:1547251570.010928:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000400:00000200:12.0:1547251570.010932:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251570.010933:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03eb0 00000400:00000200:12.0:1547251570.010936:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03ec0 00000400:00000200:13.0:1547251570.010946:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f85d [1] + 192 00000400:00000200:12.0:1547251570.010950:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f87d [1] + 192 00000100:00000200:13.0:1547251570.010955:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cae000 x1622384012443312/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251577 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251570.010968:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251570.010970:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000100:00000200:12.0:1547251570.010970:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b5700 x1622384012443328/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251577 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251570.010972:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:13.0:1547251570.010973:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:13.0:1547251570.010975:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:13.0:1547251570.010977:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:13.0:1547251570.010979:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:12.0:1547251570.010981:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251570.010982:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000100:00000200:15.0:1547251570.010983:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cae000 x1622384012443312/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251577 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251570.010983:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251570.010985:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:13.0:1547251570.010986:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251570.010986:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:12.0:1547251570.010988:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:15.0:1547251570.010989:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cae000 x1622384012443312/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251577 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251570.010989:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03ea0 00000400:00000200:15.0:1547251570.010993:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00010000:00080000:15.0:1547251570.010997:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cae000 x1622384012443312/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251577 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:1.0:1547251570.010997:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b5700 x1622384012443328/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251577 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:1.0:1547251570.011002:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b5700 x1622384012443328/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251577 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251570.011011:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00000400:00000200:13.0:1547251570.011014:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f84d [1] + 192 00010000:00080000:1.0:1547251570.011015:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b5700 x1622384012443328/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251577 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:13.0:1547251570.011018:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824756f00 x1622384012443296/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251577 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251570.011026:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251570.011027:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:13.0:1547251570.011029:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:13.0:1547251570.011029:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:8.0:1547251570.011036:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824756f00 x1622384012443296/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251577 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251570.011042:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824756f00 x1622384012443296/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251577 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251570.011046:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688440 00010000:00080000:8.0:1547251570.011049:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824756f00 x1622384012443296/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251577 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:9.0:1547251570.307659:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251570.307673:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251570.307674:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:14.0:1547251570.307682:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000400:9.0:1547251570.307682:0:12658:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.2.10.184@o2ib10: 0 seconds 00000400:00000200:9.0:1547251570.307689:0:12658:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = -110, hstatus = 6 00000800:00000200:14.0:1547251570.307692:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000400:00000200:9.0:1547251570.307692:0:12658:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.2.10.184@o2ib10: ACK: LOCAL_TIMEOUT 00000400:00000100:9.0:1547251570.307696:0:12658:0:(lib-msg.c:705:lnet_health_check()) msg 10.0.10.184@o2ib10->10.2.10.184@o2ib10 exceeded retry count 0 00000800:00000200:14.0:1547251570.308209:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:14.0:1547251570.308219:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b8244ec000 0x13e44ec000(0x13e44ec000) 00000800:00000200:14.0:1547251570.308220:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba91af4000 0x1651af4000(0x1651af4000) 00000800:00000200:14.0:1547251570.308222:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b824648000 0x13e4648000(0x13e4648000) 00000800:00000200:14.0:1547251570.308223:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97badf8e3000 0x169f8e3000(0x169f8e3000) 00000800:00000200:14.0:1547251570.308224:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b3b34a7000 0xf734a7000(0xf734a7000) 00000800:00000200:14.0:1547251570.308224:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba9dfa4000 0x165dfa4000(0x165dfa4000) 00000800:00000200:14.0:1547251570.308225:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ba9f62a000 0x165f62a000(0x165f62a000) 00000800:00000200:14.0:1547251570.308226:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97bada9b9000 0x169a9b9000(0x169a9b9000) 00000800:00000200:14.0:1547251570.308227:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b17b3d5000 0xd3b3d5000(0xd3b3d5000) 00000800:00000200:14.0:1547251570.308228:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b1d89c5000 0xd989c5000(0xd989c5000) 00000800:00000200:14.0:1547251570.308229:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ad5720c000 0x91720c000(0x91720c000) 00000800:00000200:14.0:1547251570.308230:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97af3a6e1000 0xafa6e1000(0xafa6e1000) 00000800:00000200:14.0:1547251570.308230:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97babd01d000 0x167d01d000(0x167d01d000) 00000800:00000200:14.0:1547251570.308231:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b9073ae000 0x14c73ae000(0x14c73ae000) 00000800:00000200:14.0:1547251570.308232:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97babd482000 0x167d482000(0x167d482000) 00000800:00000200:14.0:1547251570.308233:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba99f62000 0x1659f62000(0x1659f62000) 00000800:00000200:14.0:1547251570.308234:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97babd8e5000 0x167d8e5000(0x167d8e5000) 00000800:00000200:14.0:1547251570.308235:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba9298e000 0x165298e000(0x165298e000) 00000800:00000200:9.0:1547251570.308946:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308948:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308949:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308950:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308950:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308951:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308951:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308952:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308953:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308953:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308954:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308955:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308955:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308956:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308956:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308957:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308958:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308958:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308959:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308960:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308960:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308961:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308961:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308962:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308963:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308963:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308964:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308964:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308965:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308966:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308966:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308967:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308967:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308968:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:9.0:1547251570.308969:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:9.0:1547251570.308969:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ad000] (20)-- 00000800:00000100:12.0:1547251570.309286:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: don't reconnect (no need), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251570.309289:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251570.309291:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:12.0:1547251570.309294:0:70:0:(o2iblnd.c:417:kiblnd_unlink_peer_locked()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (2)-- 00000400:00000200:12.0:1547251570.309297:0:70:0:(router.c:1756:lnet_notify()) 10.1.11.90@o2ib10 notifying 10.2.10.184@o2ib10: down 00000400:00000200:12.0:1547251570.309300:0:70:0:(router.c:135:lnet_notify_locked()) set 10.2.10.184@o2ib10 0 00000800:00000200:8.2:1547251570.309338:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ad000] (19)++ 00000800:00000200:12.0:1547251570.309347:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (20)++ 00000800:00000200:12.0:1547251570.309353:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (21)-- 00000800:00000200:12.0:1547251570.309354:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (20)-- 00000800:00000200:12.0:1547251570.309355:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (19)-- 00000800:00000200:12.0:1547251570.309360:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97babe8ad000] (18)-- 00000800:00000200:8.2:1547251570.309478:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ad000] (17)++ 00000800:00000200:12.0:1547251570.309490:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (18)++ 00000800:00000200:12.0:1547251570.309494:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (19)-- 00000800:00000200:12.0:1547251570.309495:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (18)-- 00000800:00000200:12.0:1547251570.309497:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (17)++ 00000800:00000200:12.0:1547251570.309499:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (18)-- 00000800:00000200:12.0:1547251570.309501:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (17)-- 00000800:00000200:13.0:1547251570.309503:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (16)++ 00000800:00000200:13.0:1547251570.309508:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (17)-- 00000800:00000200:13.0:1547251570.309510:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (16)-- 00000800:00000200:13.0:1547251570.309511:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (15)++ 00000800:00000200:13.0:1547251570.309515:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (16)-- 00000800:00000200:13.0:1547251570.309517:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (15)-- 00000800:00000200:13.0:1547251570.309518:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (14)++ 00000800:00000200:13.0:1547251570.309521:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (15)-- 00000800:00000200:13.0:1547251570.309522:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (14)-- 00000800:00000200:13.0:1547251570.309524:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (13)++ 00000800:00000200:13.0:1547251570.309527:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (14)-- 00000800:00000200:13.0:1547251570.309528:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (13)-- 00000800:00000200:13.0:1547251570.309529:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (12)++ 00000800:00000200:13.0:1547251570.309532:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (13)-- 00000800:00000200:13.0:1547251570.309533:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (12)-- 00000800:00000200:13.0:1547251570.309534:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (11)++ 00000800:00000200:13.0:1547251570.309537:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (12)-- 00000800:00000200:13.0:1547251570.309538:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (11)-- 00000800:00000200:13.0:1547251570.309539:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (10)++ 00000800:00000200:13.0:1547251570.309543:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (11)-- 00000800:00000200:13.0:1547251570.309544:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (10)-- 00000800:00000200:13.0:1547251570.309545:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (9)++ 00000800:00000200:13.0:1547251570.309548:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (10)-- 00000800:00000200:13.0:1547251570.309549:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (9)-- 00000800:00000200:13.0:1547251570.309550:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (8)++ 00000800:00000200:13.0:1547251570.309554:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (9)-- 00000800:00000200:13.0:1547251570.309555:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (8)-- 00000800:00000200:13.0:1547251570.309556:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (7)++ 00000800:00000200:13.0:1547251570.309559:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (8)-- 00000800:00000200:13.0:1547251570.309560:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (7)-- 00000800:00000200:13.0:1547251570.309561:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (6)++ 00000800:00000200:13.0:1547251570.309564:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (7)-- 00000800:00000200:13.0:1547251570.309565:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (6)-- 00000800:00000200:13.0:1547251570.309566:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (5)++ 00000800:00000200:13.0:1547251570.309570:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (6)-- 00000800:00000200:13.0:1547251570.309570:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (5)-- 00000800:00000200:13.0:1547251570.309572:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (4)++ 00000800:00000200:13.0:1547251570.309575:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (5)-- 00000800:00000200:13.0:1547251570.309576:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (4)-- 00000800:00000200:13.0:1547251570.309577:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (3)++ 00000800:00000200:13.0:1547251570.309580:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (4)-- 00000800:00000200:13.0:1547251570.309582:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (3)-- 00000800:00000200:12.0:1547251570.309583:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ad000] (2)++ 00000800:00000200:12.0:1547251570.309593:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ad000] (3)-- 00000800:00000200:12.0:1547251570.309594:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (2)-- 00000800:00000200:12.0:1547251570.309596:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ad000] (1)-- 00000800:00000200:9.0:1547251570.310409:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97a59c828a80] -> 10.2.10.184@o2ib10 (1)-- 00000100:00000200:9.0:1547251572.122682:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443344, portal 4 00000100:00000200:9.0:1547251572.122684:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443344, offset 0 00000400:00000200:9.0:1547251572.122688:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251572.122690:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251572.122692:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23000] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251572.122696:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22995] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23000] 00000400:00000200:9.0:1547251572.122698:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000100:00000200:1.0:1547251572.122698:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443360, portal 4 00000400:00000200:9.0:1547251572.122699:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:1.0:1547251572.122700:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443360, offset 0 00000400:00000200:9.0:1547251572.122701:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251572.122702:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9734, 9733] 00000400:00000200:9.0:1547251572.122704:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:1.0:1547251572.122704:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:1.0:1547251572.122706:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251572.122708:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000400:00000200:1.0:1547251572.122709:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23000] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:9.0:1547251572.122711:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:1.0:1547251572.122712:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22996] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23000] 00000800:00000200:9.0:1547251572.122714:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251572.122715:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000400:00000200:1.0:1547251572.122715:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:9.0:1547251572.122716:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:1.0:1547251572.122716:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251572.122717:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251572.122719:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:1.0:1547251572.122719:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9734, 9734] 00000400:00000200:1.0:1547251572.122721:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:1.0:1547251572.122725:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:1.0:1547251572.122728:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:1.0:1547251572.122731:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251572.122732:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:1.0:1547251572.122733:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000800:00000200:1.0:1547251572.122736:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)-- 00000100:00000200:13.0:1547251572.122737:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443376, portal 4 00000100:00000200:13.0:1547251572.122741:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443376, offset 0 00000400:00000200:13.0:1547251572.122745:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:13.0:1547251572.122748:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:13.0:1547251572.122750:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23001] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:13.0:1547251572.122753:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22996] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23001] 00000400:00000200:13.0:1547251572.122756:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:13.0:1547251572.122757:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251572.122758:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251572.122760:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9735, 9734] 00000400:00000200:13.0:1547251572.122762:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:13.0:1547251572.122766:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251572.122769:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:7.2:1547251572.122769:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:13.0:1547251572.122772:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251572.122773:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:13.0:1547251572.122774:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:13.0:1547251572.122777:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (24)-- 00000800:00000200:5.2:1547251572.122791:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251572.122803:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:10.0:1547251572.122807:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (24)-- 00000800:00000200:11.0:1547251572.122808:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251572.122811:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000100:00000200:10.0:1547251572.122812:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf0000 x1622384012443344/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251572.122814:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b5a00 x1622384012443360/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251572.122819:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea7f8 00000400:00000200:10.0:1547251572.122820:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251572.122821:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2a18 00000400:00000200:11.0:1547251572.122822:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251572.122824:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251572.122825:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251572.122827:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:11.0:1547251572.122828:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251572.122829:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:11.0:1547251572.122830:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:10.0:1547251572.122830:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:11.0:1547251572.122832:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000100:00000200:10.0:1547251572.122833:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a6000 x1622384012443376/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251572.122838:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792bd48 00000400:00000200:10.0:1547251572.122839:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251572.122843:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251572.122844:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:5.2:1547251572.122892:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251572.122897:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:7.2:1547251572.122903:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:11.0:1547251572.122906:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251572.122910:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251572.122910:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251572.122913:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251572.122915:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251572.122917:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251572.122919:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03ed0 00000400:00000200:10.0:1547251572.122921:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03ef0 00000400:00000200:11.0:1547251572.122924:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6819 [1] + 192 00000400:00000200:10.0:1547251572.122929:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6849 [1] + 192 00000100:00000200:11.0:1547251572.122933:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf0000 x1622384012443344/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251572.122954:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251572.122956:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a6000 x1622384012443376/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251572.122957:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251572.122959:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251572.122961:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251572.122962:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251572.122965:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251572.122967:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:11.0:1547251572.122969:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251572.122969:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000100:00000200:9.0:1547251572.122969:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf0000 x1622384012443344/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251572.122970:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251572.122971:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:11.0:1547251572.122973:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03ee0 00000800:00000200:10.0:1547251572.122973:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:9.0:1547251572.122974:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf0000 x1622384012443344/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251572.122975:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:10.0:1547251572.122976:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:9.0:1547251572.122978:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea2a8 00000100:00000200:13.0:1547251572.122982:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a6000 x1622384012443376/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251572.122982:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf0000 x1622384012443344/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:13.0:1547251572.122987:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a6000 x1622384012443376/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251572.122992:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b2a8 00000400:00000200:11.0:1547251572.122994:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6839 [1] + 192 00010000:00080000:13.0:1547251572.122995:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a6000 x1622384012443376/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251572.122998:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b5a00 x1622384012443360/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251572.123006:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251572.123007:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251572.123008:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (22)-- 00000800:00000200:11.0:1547251572.123009:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:1.0:1547251572.123016:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b5a00 x1622384012443360/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:1.0:1547251572.123022:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b5a00 x1622384012443360/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251572.123027:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00010000:00080000:1.0:1547251572.123030:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b5a00 x1622384012443360/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251572.586679:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443392, portal 4 00000100:00000200:8.0:1547251572.586682:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443392, offset 0 00000400:00000200:8.0:1547251572.586685:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251572.586689:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251572.586691:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23001] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:1.0:1547251572.586693:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443408, portal 4 00000400:00000200:8.0:1547251572.586694:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22997] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23001] 00000400:00000200:8.0:1547251572.586696:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000100:00000200:1.0:1547251572.586696:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443408, offset 0 00000400:00000200:8.0:1547251572.586698:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251572.586699:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251572.586700:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251572.586701:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9726, 9728] 00000400:00000200:8.0:1547251572.586702:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:1.0:1547251572.586703:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:1.0:1547251572.586705:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23001] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251572.586707:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000100:00000200:12.0:1547251572.586709:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443424, portal 4 00000400:00000200:1.0:1547251572.586709:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22998] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23001] 00000800:00000200:8.0:1547251572.586710:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:1.0:1547251572.586711:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000100:00000200:12.0:1547251572.586713:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443424, offset 0 00000800:00000200:8.0:1547251572.586713:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:1.0:1547251572.586713:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251572.586714:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000400:00000200:1.0:1547251572.586714:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251572.586715:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:1.0:1547251572.586715:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9727, 9728] 00000400:00000200:12.0:1547251572.586716:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:1.0:1547251572.586717:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:8.0:1547251572.586718:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:12.0:1547251572.586719:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:12.0:1547251572.586722:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23002] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251572.586722:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:12.0:1547251572.586725:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:22998] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23002] 00000800:00000200:1.0:1547251572.586725:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:12.0:1547251572.586728:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:12.0:1547251572.586729:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:1.0:1547251572.586729:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:12.0:1547251572.586731:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:1.0:1547251572.586731:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000400:00000200:12.0:1547251572.586732:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9727, 9729] 00000800:00000200:1.0:1547251572.586733:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000400:00000200:12.0:1547251572.586734:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:1.0:1547251572.586737:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:0.2:1547251572.586737:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:12.0:1547251572.586739:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251572.586741:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:12.2:1547251572.586745:0:12682:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:12.0:1547251572.586759:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251572.586760:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:12.0:1547251572.586761:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (23)++ 00000800:00000200:12.0:1547251572.586764:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:5.0:1547251572.586793:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)++ 00000800:00000200:5.0:1547251572.586796:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:4.0:1547251572.586797:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251572.586799:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000100:00000200:5.0:1547251572.586801:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824754500 x1622384012443392/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251572.586802:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b5d00 x1622384012443408/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251572.586808:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000400:00000200:4.0:1547251572.586809:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e22a8 00000400:00000200:5.0:1547251572.586810:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251572.586810:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251572.586814:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251572.586814:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251572.586817:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251572.586818:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251572.586819:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251572.586820:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:4.0:1547251572.586820:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251572.586821:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:4.0:1547251572.586823:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f0600 x1622384012443424/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251572.586829:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038990 00000400:00000200:4.0:1547251572.586830:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251572.586833:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251572.586835:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:0.2:1547251572.586889:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251572.586896:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:5.0:1547251572.586905:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251572.586909:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251572.586909:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251572.586912:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251572.586914:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251572.586917:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251572.586919:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03f00 00000400:00000200:4.0:1547251572.586921:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03f10 00000400:00000200:5.0:1547251572.586923:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3575 [1] + 192 00000400:00000200:4.0:1547251572.586928:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3585 [1] + 192 00000100:00000200:5.0:1547251572.586932:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824754500 x1622384012443392/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251572.586953:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251572.586955:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b5d00 x1622384012443408/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251572.586956:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251572.586958:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251572.586959:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251572.586961:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251572.586964:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251572.586966:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251572.586968:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824754500 x1622384012443392/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251572.586968:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251572.586968:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251572.586970:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000400:00000200:5.0:1547251572.586971:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03f20 00000800:00000200:4.0:1547251572.586971:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251572.586973:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:8.0:1547251572.586974:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824754500 x1622384012443392/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251572.586974:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:8.0:1547251572.586978:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688440 00010000:00080000:8.0:1547251572.586981:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824754500 x1622384012443392/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251572.586981:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3595 [1] + 192 00000100:00000200:1.0:1547251572.586986:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b5d00 x1622384012443408/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:1.0:1547251572.586992:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b5d00 x1622384012443408/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251572.586996:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00000100:00000200:5.0:1547251572.586999:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f0600 x1622384012443424/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:1.0:1547251572.587000:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b5d00 x1622384012443408/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251572.587008:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251572.587009:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251572.587010:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251572.587011:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:12.0:1547251572.587016:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f0600 x1622384012443424/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251572.587022:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f0600 x1622384012443424/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251572.587026:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038cc0 00010000:00080000:12.0:1547251572.587029:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f0600 x1622384012443424/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251572.986673:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443440, portal 4 00000100:00000200:8.0:1547251572.986676:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443440, offset 0 00000400:00000200:8.0:1547251572.986679:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251572.986681:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251572.986684:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03f30 00000400:00000200:8.0:1547251572.986686:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4179 [1] + 0 00000400:00000200:8.0:1547251572.986689:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d6883b8 00000100:00000200:8.0:1547251572.986690:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251572.986699:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6883b8 00000400:00000200:8.0:1547251572.986700:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251572.986705:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824750f00 x1622384012443440/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251572.986712:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00000400:00000200:8.0:1547251572.986713:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251572.986714:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251572.986721:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443440 00010000:00000200:9.0:1547251572.986731:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3dd450 x1622384012443440/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:633/0 lens 224/368 e 0 to 0 dl 1547251578 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:7.0:1547251572.986732:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443456, portal 4 00000100:00000200:7.0:1547251572.986735:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443456, offset 0 00000400:00000200:7.0:1547251572.986737:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:9.0:1547251572.986738:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443440, offset 192 00000400:00000200:9.0:1547251572.986740:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251572.986740:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251572.986743:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251572.986743:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03f40 00000400:00000200:9.0:1547251572.986746:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03f30 00000400:00000200:7.0:1547251572.986746:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1da5 [1] + 0 00000400:00000200:9.0:1547251572.986748:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaeab1 [1] + 192 00000400:00000200:7.0:1547251572.986749:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990550 00000100:00000200:7.0:1547251572.986750:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:9.0:1547251572.986751:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824750f00 x1622384012443440/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251572.986758:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990550 00000400:00000200:9.0:1547251572.986759:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251572.986760:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251572.986761:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea2a8 00000400:00000200:9.0:1547251572.986762:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:7.0:1547251572.986764:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df6900 x1622384012443456/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251572.986765:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251572.986769:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824750f00 x1622384012443440/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251572.986770:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000100:00000200:9.0:1547251572.986771:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:7.0:1547251572.986771:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251572.986773:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251572.986775:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824750f00 x1622384012443440/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251572.986781:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688440 00000100:00000200:6.0:1547251572.986783:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443456 00010000:00080000:8.0:1547251572.986785:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824750f00 x1622384012443440/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/0 rc 0/0 00010000:00000200:6.0:1547251572.986793:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c118450 x1622384012443456/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:633/0 lens 224/368 e 0 to 0 dl 1547251578 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251572.986799:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443456, offset 192 00000400:00000200:6.0:1547251572.986802:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251572.986804:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251572.986807:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03f40 00000400:00000200:6.0:1547251572.986809:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaeac1 [1] + 192 00000100:00000200:6.0:1547251572.986812:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df6900 x1622384012443456/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251572.986821:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251572.986823:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459904c8 00000400:00000200:6.0:1547251572.986824:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251572.986826:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251572.986831:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df6900 x1622384012443456/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251572.986833:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251572.986837:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df6900 x1622384012443456/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251572.986842:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00010000:00080000:7.0:1547251572.986846:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df6900 x1622384012443456/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251579 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251573.002678:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443472, portal 4 00000100:00000200:7.0:1547251573.002681:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443472, offset 0 00000400:00000200:7.0:1547251573.002684:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251573.002687:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251573.002690:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03f50 00000400:00000200:7.0:1547251573.002692:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1dd5 [1] + 0 00000400:00000200:7.0:1547251573.002694:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990000 00000100:00000200:7.0:1547251573.002696:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:7.0:1547251573.002704:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990000 00000400:00000200:7.0:1547251573.002705:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251573.002709:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df2d00 x1622384012443472/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251580 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251573.002716:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:7.0:1547251573.002717:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251573.002718:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251573.002727:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443472 00010000:00000200:6.0:1547251573.002737:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c118050 x1622384012443472/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:634/0 lens 224/368 e 0 to 0 dl 1547251579 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251573.002744:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443472, offset 192 00000400:00000200:6.0:1547251573.002746:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251573.002749:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251573.002751:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03f50 00000400:00000200:6.0:1547251573.002753:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaead1 [1] + 192 00000100:00000200:6.0:1547251573.002756:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df2d00 x1622384012443472/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251580 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251573.002764:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251573.002768:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459906e8 00000400:00000200:6.0:1547251573.002770:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251573.002772:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251573.002775:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df2d00 x1622384012443472/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251580 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251573.002778:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251573.002780:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df2d00 x1622384012443472/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251580 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251573.002786:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00010000:00080000:7.0:1547251573.002790:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df2d00 x1622384012443472/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251580 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:0.0:1547251575.018682:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443488, portal 4 00000100:00000200:0.0:1547251575.018685:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443488, offset 0 00000400:00000200:0.0:1547251575.018689:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:0.0:1547251575.018692:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:0.0:1547251575.018694:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23002] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251575.018697:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:22999] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23002] 00000400:00000200:0.0:1547251575.018700:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251575.018701:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251575.018703:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251575.018704:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9729, 9729] 00000400:00000200:0.0:1547251575.018706:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000100:00000200:9.0:1547251575.018708:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443504, portal 4 00000100:00000200:15.0:1547251575.018709:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443520, portal 4 00000100:00000200:9.0:1547251575.018710:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443504, offset 0 00000400:00000200:0.0:1547251575.018710:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000100:00000200:15.0:1547251575.018712:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443520, offset 0 00000400:00000200:9.0:1547251575.018713:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:0.0:1547251575.018713:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:15.0:1547251575.018716:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:9.0:1547251575.018716:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000800:00000200:0.0:1547251575.018716:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251575.018717:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:0.0:1547251575.018718:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:15.0:1547251575.018719:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:9.0:1547251575.018719:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23002] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251575.018721:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23002] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:0.0:1547251575.018721:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:9.0:1547251575.018722:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23000] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23002] 00000400:00000200:9.0:1547251575.018724:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:15.0:1547251575.018725:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23000] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23002] 00000400:00000200:15.0:1547251575.018727:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:9.2:1547251575.018728:0:12680:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:15.0:1547251575.018729:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251575.018730:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251575.018731:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9729, 9730] 00000400:00000200:15.0:1547251575.018733:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251575.018738:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251575.018740:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:9.0:1547251575.018740:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251575.018742:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:15.0:1547251575.018743:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000400:00000200:9.0:1547251575.018743:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9730, 9730] 00000400:00000200:9.0:1547251575.018745:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000800:00000200:15.0:1547251575.018746:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:15.0:1547251575.018747:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:15.0:1547251575.018750:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:9.0:1547251575.018750:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251575.018752:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:12.0:1547251575.018755:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:9.0:1547251575.018755:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251575.018757:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.0:1547251575.018758:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:9.0:1547251575.018760:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251575.018767:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:12.0:1547251575.018771:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328dc200 x1622384012443488/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251582 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251575.018776:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:12.0:1547251575.018779:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665990 00000400:00000200:12.0:1547251575.018780:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251575.018783:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251575.018785:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:7.2:1547251575.018799:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:7.2:1547251575.018809:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000800:00000200:14.0:1547251575.018815:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:14.0:1547251575.018816:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:14.0:1547251575.018820:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cabc00 x1622384012443520/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251582 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251575.018821:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:12.0:1547251575.018823:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000400:00000200:14.0:1547251575.018825:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1d48 00000100:00000200:12.0:1547251575.018825:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf4800 x1622384012443504/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251582 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251575.018826:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251575.018828:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:12.0:1547251575.018830:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea3b8 00000800:00000200:14.0:1547251575.018831:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:12.0:1547251575.018831:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:14.0:1547251575.018832:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000400:00000200:12.0:1547251575.018833:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251575.018834:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251575.018835:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:7.2:1547251575.018888:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251575.018893:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:14.0:1547251575.018900:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:14.0:1547251575.018902:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251575.018905:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:14.0:1547251575.018906:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251575.018907:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000800:00000200:7.2:1547251575.018907:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000400:00000200:14.0:1547251575.018909:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03f70 00000400:00000200:12.0:1547251575.018911:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251575.018912:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f8bd [1] + 192 00000400:00000200:12.0:1547251575.018915:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03f60 00000100:00000200:14.0:1547251575.018917:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf4800 x1622384012443504/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251582 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251575.018926:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251575.018928:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000400:00000200:12.0:1547251575.018928:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f8ad [1] + 192 00000800:00000200:14.0:1547251575.018929:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:14.0:1547251575.018930:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:14.0:1547251575.018931:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:12.0:1547251575.018931:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328dc200 x1622384012443488/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251582 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251575.018933:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:14.0:1547251575.018934:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:14.0:1547251575.018936:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251575.018947:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251575.018947:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251575.018949:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (21)++ 00000100:00000200:9.0:1547251575.018949:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf4800 x1622384012443504/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251582 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251575.018950:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03f80 00000800:00000200:12.0:1547251575.018951:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251575.018952:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:12.0:1547251575.018953:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:9.0:1547251575.018955:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf4800 x1622384012443504/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251582 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251575.018959:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ead48 00000400:00000200:14.0:1547251575.018962:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f8cd [1] + 192 00010000:00080000:9.0:1547251575.018963:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf4800 x1622384012443504/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251582 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:0.0:1547251575.018966:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328dc200 x1622384012443488/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251582 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251575.018971:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328dc200 x1622384012443488/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251582 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251575.018975:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00000100:00000200:14.0:1547251575.018977:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cabc00 x1622384012443520/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251582 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251575.018979:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328dc200 x1622384012443488/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251582 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:14.0:1547251575.018985:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251575.018986:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:14.0:1547251575.018988:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:14.0:1547251575.018989:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000100:00000200:15.0:1547251575.018997:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cabc00 x1622384012443520/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251582 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:15.0:1547251575.019004:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cabc00 x1622384012443520/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251582 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251575.019010:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00010000:00080000:15.0:1547251575.019015:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cabc00 x1622384012443520/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251582 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:0.0:1547251577.130677:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443536, portal 4 00000100:00000200:0.0:1547251577.130680:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443536, offset 0 00000400:00000200:0.0:1547251577.130683:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:0.0:1547251577.130686:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:0.0:1547251577.130688:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23004] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251577.130691:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23000] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23004] 00000400:00000200:0.0:1547251577.130694:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251577.130695:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251577.130698:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251577.130699:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9735, 9735] 00000400:00000200:0.0:1547251577.130701:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:0.0:1547251577.130705:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000100:00000200:12.0:1547251577.130706:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443552, portal 4 00000800:00000200:0.0:1547251577.130708:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000100:00000200:12.0:1547251577.130709:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443552, offset 0 00000100:00000200:9.0:1547251577.130709:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443568, portal 4 00000800:00000200:0.0:1547251577.130711:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000400:00000200:12.0:1547251577.130712:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000100:00000200:9.0:1547251577.130712:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443568, offset 0 00000800:00000200:0.0:1547251577.130713:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:0.0:1547251577.130714:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:12.0:1547251577.130715:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251577.130715:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:0.0:1547251577.130716:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:12.0:1547251577.130717:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23004] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251577.130718:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:12.0:1547251577.130720:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23001] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23004] 00000400:00000200:9.0:1547251577.130720:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23004] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251577.130723:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251577.130723:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23001] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23004] 00000400:00000200:12.0:1547251577.130725:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251577.130726:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:12.0:1547251577.130727:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251577.130727:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251577.130728:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9735, 9736] 00000400:00000200:9.0:1547251577.130728:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251577.130730:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:9.0:1547251577.130730:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9735, 9736] 00000400:00000200:9.0:1547251577.130732:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:9.0:1547251577.130736:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251577.130739:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:12.0:1547251577.130742:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251577.130745:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:9.0:1547251577.130746:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251577.130747:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:12.0:1547251577.130748:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251577.130748:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000800:00000200:12.0:1547251577.130750:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:12.0:1547251577.130751:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (23)++ 00000800:00000200:9.0:1547251577.130751:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:12.0:1547251577.130754:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:7.2:1547251577.130765:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251577.130790:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251577.130799:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:5.2:1547251577.130801:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (22)++ 00000100:00000200:10.0:1547251577.130804:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328dc500 x1622384012443536/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251577.130807:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251577.130809:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (23)++ 00000800:00000200:11.0:1547251577.130811:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (24)-- 00000400:00000200:10.0:1547251577.130812:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665220 00000400:00000200:10.0:1547251577.130813:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:11.0:1547251577.130815:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf4b00 x1622384012443568/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251577.130817:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251577.130819:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:10.0:1547251577.130821:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (23)++ 00000400:00000200:11.0:1547251577.130822:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea4c8 00000800:00000200:10.0:1547251577.130822:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (24)-- 00000400:00000200:11.0:1547251577.130823:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:10.0:1547251577.130825:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f0900 x1622384012443552/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251577.130826:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251577.130828:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251577.130830:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:10.0:1547251577.130830:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038bb0 00000400:00000200:10.0:1547251577.130831:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251577.130834:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251577.130836:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:5.2:1547251577.130911:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251577.130917:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:5.2:1547251577.130924:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:11.0:1547251577.130927:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251577.130931:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251577.130931:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251577.130934:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251577.130935:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251577.130947:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251577.130949:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03f90 00000400:00000200:10.0:1547251577.130951:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03fb0 00000400:00000200:11.0:1547251577.130953:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd68a9 [1] + 192 00000400:00000200:10.0:1547251577.130958:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd68c9 [1] + 192 00000100:00000200:11.0:1547251577.130961:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328dc500 x1622384012443536/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251577.130973:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251577.130975:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf4b00 x1622384012443568/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251577.130977:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251577.130979:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251577.130980:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251577.130982:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251577.130984:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251577.130986:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251577.130987:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000100:00000200:0.0:1547251577.130988:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328dc500 x1622384012443536/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251577.130989:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251577.130989:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251577.130990:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:11.0:1547251577.130992:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03fa0 00000800:00000200:10.0:1547251577.130992:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:10.0:1547251577.130994:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:0.0:1547251577.130994:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328dc500 x1622384012443536/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251577.130995:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:0.0:1547251577.130998:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00000100:00000200:9.0:1547251577.131001:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf4b00 x1622384012443568/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251577.131002:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328dc500 x1622384012443536/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251577.131007:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf4b00 x1622384012443568/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251577.131011:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ead48 00000400:00000200:11.0:1547251577.131014:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd68b9 [1] + 192 00010000:00080000:9.0:1547251577.131014:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf4b00 x1622384012443568/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251577.131017:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f0900 x1622384012443552/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251577.131025:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251577.131026:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251577.131027:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251577.131028:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:12.0:1547251577.131081:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f0900 x1622384012443552/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251577.131087:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f0900 x1622384012443552/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251577.131092:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038cc0 00010000:00080000:12.0:1547251577.131095:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f0900 x1622384012443552/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:12.0:1547251577.594693:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443600, portal 4 00000100:00000200:12.0:1547251577.594696:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443600, offset 0 00000100:00000200:0.0:1547251577.594699:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443584, portal 4 00000400:00000200:12.0:1547251577.594700:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000100:00000200:0.0:1547251577.594702:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443584, offset 0 00000400:00000200:12.0:1547251577.594703:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:12.0:1547251577.594705:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23006] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251577.594705:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:12.0:1547251577.594708:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23001] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23006] 00000400:00000200:0.0:1547251577.594708:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000100:00000200:9.0:1547251577.594709:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443616, portal 4 00000400:00000200:12.0:1547251577.594711:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251577.594711:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23006] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251577.594712:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:9.0:1547251577.594712:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443616, offset 0 00000400:00000200:12.0:1547251577.594714:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251577.594714:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23002] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23006] 00000400:00000200:12.0:1547251577.594715:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9728, 9729] 00000400:00000200:9.0:1547251577.594715:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:0.0:1547251577.594716:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:12.0:1547251577.594717:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:9.0:1547251577.594718:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:0.0:1547251577.594718:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251577.594720:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251577.594721:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23006] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251577.594721:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9729, 9729] 00000400:00000200:12.0:1547251577.594722:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:0.0:1547251577.594723:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:9.0:1547251577.594724:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23003] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23006] 00000800:00000200:12.0:1547251577.594725:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:9.0:1547251577.594726:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:12.0:1547251577.594728:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:9.0:1547251577.594728:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251577.594728:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:9.0:1547251577.594729:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:12.0:1547251577.594730:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.0:1547251577.594731:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:9.0:1547251577.594731:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9729, 9730] 00000800:00000200:0.0:1547251577.594731:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251577.594733:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:12.0:1547251577.594734:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:0.0:1547251577.594734:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:9.0:1547251577.594737:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251577.594737:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:0.0:1547251577.594738:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:9.0:1547251577.594740:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:9.0:1547251577.594743:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:0.2:1547251577.594743:0:12671:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:9.0:1547251577.594744:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:9.0:1547251577.594745:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:9.0:1547251577.594748:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:0.2:1547251577.594751:0:12671:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:0.0:1547251577.594761:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:12.2:1547251577.594764:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251577.594801:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:5.0:1547251577.594804:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:4.0:1547251577.594805:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251577.594807:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:5.0:1547251577.594809:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328dc800 x1622384012443584/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251577.594810:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f0c00 x1622384012443600/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251577.594816:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665a18 00000400:00000200:4.0:1547251577.594816:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038220 00000400:00000200:5.0:1547251577.594818:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251577.594818:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251577.594822:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251577.594822:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251577.594825:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251577.594826:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251577.594827:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251577.594828:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000800:00000200:4.0:1547251577.594828:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251577.594829:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:5.0:1547251577.594831:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf4e00 x1622384012443616/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251577.594831:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:5.0:1547251577.594837:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eabb0 00000400:00000200:5.0:1547251577.594839:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251577.594842:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251577.594844:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:12.2:1547251577.594907:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:0.2:1547251577.594919:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:4.0:1547251577.594926:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:12.2:1547251577.594928:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:4.0:1547251577.594930:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000800:00000200:5.0:1547251577.594933:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:4.0:1547251577.594935:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:5.0:1547251577.594936:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251577.594951:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03fd0 00000400:00000200:5.0:1547251577.594953:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251577.594956:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3605 [1] + 192 00000400:00000200:5.0:1547251577.594957:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03fe0 00000400:00000200:5.0:1547251577.594960:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3625 [1] + 192 00000100:00000200:4.0:1547251577.594964:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f0c00 x1622384012443600/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251577.594981:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251577.594983:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf4e00 x1622384012443616/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251577.594985:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251577.594987:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:4.0:1547251577.594988:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251577.594990:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:4.0:1547251577.594992:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000400:00000200:5.0:1547251577.594994:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251577.594994:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000100:00000200:12.0:1547251577.594996:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f0c00 x1622384012443600/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251577.594996:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251577.594997:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251577.594999:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:4.0:1547251577.594999:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:5.0:1547251577.595000:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:12.0:1547251577.595002:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f0c00 x1622384012443600/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251577.595002:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c03fc0 00000800:00000200:5.0:1547251577.595003:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:12.0:1547251577.595006:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038cc0 00010000:00080000:12.0:1547251577.595009:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f0c00 x1622384012443600/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251577.595009:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf4e00 x1622384012443616/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251577.595014:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf4e00 x1622384012443616/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251577.595019:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ead48 00010000:00080000:9.0:1547251577.595022:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf4e00 x1622384012443616/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:4.0:1547251577.595022:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3615 [1] + 192 00000100:00000200:4.0:1547251577.595026:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328dc800 x1622384012443584/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251577.595035:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251577.595036:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251577.595037:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251577.595038:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:0.0:1547251577.595045:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328dc800 x1622384012443584/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251577.595050:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328dc800 x1622384012443584/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251577.595055:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00010000:00080000:0.0:1547251577.595058:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328dc800 x1622384012443584/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251577.994692:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443648, portal 4 00000100:00000200:6.0:1547251577.994693:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443632, portal 4 00000100:00000200:9.0:1547251577.994695:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443648, offset 0 00000100:00000200:6.0:1547251577.994696:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443632, offset 0 00000400:00000200:9.0:1547251577.994698:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251577.994698:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251577.994701:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251577.994701:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251577.994704:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04000 00000400:00000200:6.0:1547251577.994704:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c03ff0 00000400:00000200:9.0:1547251577.994706:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4269 [1] + 0 00000400:00000200:6.0:1547251577.994707:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1e55 [1] + 0 00000400:00000200:9.0:1547251577.994708:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ae6a914440 00000400:00000200:6.0:1547251577.994709:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17ea440 00000100:00000200:9.0:1547251577.994710:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:6.0:1547251577.994711:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251577.994718:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914440 00000400:00000200:9.0:1547251577.994719:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251577.994724:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf5100 x1622384012443648/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251577.994724:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea440 00000400:00000200:6.0:1547251577.994725:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251577.994728:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37935d00 x1622384012443632/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251577.994731:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea440 00000400:00000200:9.0:1547251577.994732:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251577.994733:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990880 00000400:00000200:6.0:1547251577.994733:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251577.994734:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251577.994735:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251577.994740:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443648 00000100:00000200:4.0:1547251577.994747:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443632 00010000:00000200:8.0:1547251577.994750:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3dd050 x1622384012443648/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:638/0 lens 224/368 e 0 to 0 dl 1547251583 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:8.0:1547251577.994757:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443648, offset 192 00000400:00000200:8.0:1547251577.994759:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00010000:00000200:4.0:1547251577.994760:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39fc50 x1622384012443632/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:638/0 lens 224/368 e 0 to 0 dl 1547251583 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:8.0:1547251577.994762:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251577.994765:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04000 00000400:00000200:8.0:1547251577.994767:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaeb11 [1] + 192 00000100:00000200:4.0:1547251577.994769:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443632, offset 192 00000100:00000200:8.0:1547251577.994770:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf5100 x1622384012443648/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251577.994772:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251577.994776:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251577.994778:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251577.994779:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c03ff0 00000400:00000200:8.0:1547251577.994780:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688440 00000400:00000200:8.0:1547251577.994781:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251577.994782:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaeb21 [1] + 192 00000400:00000200:8.0:1547251577.994784:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251577.994785:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37935d00 x1622384012443632/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251577.994790:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:4.0:1547251577.994795:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251577.994797:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf5100 x1622384012443648/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251577.994797:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b660 00000400:00000200:4.0:1547251577.994798:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251577.994799:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251577.994804:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf5100 x1622384012443648/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251577.994805:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:9.0:1547251577.994809:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ead48 00000100:00000200:6.0:1547251577.994812:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37935d00 x1622384012443632/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251577.994813:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf5100 x1622384012443648/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251577.994817:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37935d00 x1622384012443632/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251577.994821:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990220 00010000:00080000:6.0:1547251577.994824:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37935d00 x1622384012443632/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251584 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251578.010675:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443664, portal 4 00000100:00000200:6.0:1547251578.010678:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443664, offset 0 00000400:00000200:6.0:1547251578.010680:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251578.010683:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251578.010686:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04010 00000400:00000200:6.0:1547251578.010688:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1e85 [1] + 0 00000400:00000200:6.0:1547251578.010690:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17ea220 00000100:00000200:6.0:1547251578.010693:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251578.010702:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea220 00000400:00000200:6.0:1547251578.010703:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251578.010705:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37936000 x1622384012443664/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251585 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251578.010710:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990880 00000400:00000200:6.0:1547251578.010710:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251578.010712:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251578.010733:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443664 00010000:00000200:5.0:1547251578.010742:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39f850 x1622384012443664/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:639/0 lens 224/368 e 0 to 0 dl 1547251584 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251578.010750:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443664, offset 192 00000400:00000200:5.0:1547251578.010752:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251578.010755:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251578.010758:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04010 00000400:00000200:5.0:1547251578.010760:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaeb31 [1] + 192 00000100:00000200:5.0:1547251578.010764:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37936000 x1622384012443664/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251585 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251578.010773:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251578.010775:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad3b8 00000400:00000200:5.0:1547251578.010776:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251578.010777:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251578.010780:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37936000 x1622384012443664/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251585 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251578.010782:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251578.010785:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37936000 x1622384012443664/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251585 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251578.010789:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990220 00010000:00080000:6.0:1547251578.010792:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37936000 x1622384012443664/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251585 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:0.2:1547251579.741374:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:12.2:1547251579.741383:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:4.0:1547251579.741431:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251579.741435:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251579.741436:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000800:00000200:5.0:1547251579.741439:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251579.741442:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251579.741443:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251579.741449:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b10270 00000400:00000200:4.0:1547251579.741449:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b10260 00000400:00000200:4.0:1547251579.741453:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 58240 00000400:00000200:5.0:1547251579.741458:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 58464 00000100:00000200:4.0:1547251579.741461:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:5.0:1547251579.741472:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:4.0:1547251579.741472:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251579.741474:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251579.741476:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000400:00000200:5.0:1547251579.741477:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251579.741478:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251579.741479:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251579.741479:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251579.741481:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251579.741482:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251579.741482:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000800:00000200:5.0:1547251579.741484:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251579.741486:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251579.741486:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251579.741490:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251579.741490:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b10250 00000400:00000200:4.0:1547251579.741493:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 58688 00000400:00000200:5.0:1547251579.741494:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 26 MB=0x5c38cb6b10240 00000100:00000200:4.0:1547251579.741496:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:5.0:1547251579.741497:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0xd4af9 [1] + 0 00000400:00000200:4.0:1547251579.741499:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251579.741500:0:12662:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d440 00000800:00000200:4.0:1547251579.741501:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251579.741502:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000100:00000200:5.0:1547251579.741503:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000800:00000200:4.0:1547251579.741503:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251579.741505:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251579.741507:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:5.0:1547251579.741511:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d440 00000400:00000200:5.0:1547251579.741512:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251579.741513:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251579.741515:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251579.741516:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000100:00000200:4.0:1547251579.741523:0:12979:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444832 00010000:00000200:4.0:1547251579.741529:0:12979:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444b000 x1622384011444832/t0(0) o400->cache1-MDT0000-lwp-OST0006_UUID@10.0.11.92@o2ib10:640/0 lens 224/224 e 0 to 0 dl 1547251585 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:6.0:1547251579.741530:0:13355:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444848 00000100:00000200:5.0:1547251579.741532:0:12981:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444816 00010000:00000200:6.0:1547251579.741536:0:13355:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae5c432d00 x1622384011444848/t0(0) o400->cache1-MDT0000-lwp-OST0008_UUID@10.0.11.92@o2ib10:640/0 lens 224/224 e 0 to 0 dl 1547251585 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:5.0:1547251579.741536:0:12981:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444f800 x1622384011444816/t0(0) o400->cache1-MDT0000-lwp-OST0007_UUID@10.0.11.92@o2ib10:640/0 lens 224/224 e 0 to 0 dl 1547251585 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:4.0:1547251579.741538:0:12979:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444832, offset 192 00000400:00000200:4.0:1547251579.741542:0:12979:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000100:00000200:6.0:1547251579.741544:0:13355:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444848, offset 192 00000100:00000200:5.0:1547251579.741544:0:12981:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444816, offset 192 00000400:00000200:4.0:1547251579.741546:0:12979:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:6.0:1547251579.741547:0:13355:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000400:00000200:4.0:1547251579.741549:0:12979:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:5.0:1547251579.741550:0:12981:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:4.0:1547251579.741551:0:12979:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251579.741553:0:12979:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9730, 9730] 00000400:00000200:4.0:1547251579.741555:0:12979:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251579.741562:0:12979:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:4.0:1547251579.741565:0:12979:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:6.0:1547251579.741566:0:13355:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00000400:00000200:6.0:1547251579.741569:0:13355:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251579.741570:0:13355:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:14.0:1547251579.741571:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444800 00000400:00000200:6.0:1547251579.741572:0:13355:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9730, 9731] 00000400:00000200:6.0:1547251579.741573:0:13355:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00010000:00000200:14.0:1547251579.741577:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2db450 x1622384011444800/t0(0) o400->04d65b15-cf9c-7ded-611d-ffe7fe3f3abf@10.0.11.92@o2ib10:640/0 lens 224/224 e 0 to 0 dl 1547251585 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:6.0:1547251579.741579:0:13355:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.1.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:6.0:1547251579.741581:0:13355:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:5.0:1547251579.741583:0:12981:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000100:00000200:14.0:1547251579.741592:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384011444800, offset 192 00000400:00000200:14.0:1547251579.741595:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000400:00000200:14.0:1547251579.741599:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00000400:00000200:14.0:1547251579.741601:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:5.0:1547251579.741602:0:12981:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251579.741603:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251579.741604:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9731, 9731] 00000400:00000200:5.0:1547251579.741604:0:12981:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251579.741606:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251579.741606:0:12981:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9731, 9731] 00000400:00000200:5.0:1547251579.741609:0:12981:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251579.741616:0:12981:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:5.0:1547251579.741619:0:12981:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:6.0:1547251579.741624:0:13355:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:4.0:1547251579.741624:0:12979:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:6.0:1547251579.741625:0:13355:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:4.0:1547251579.741625:0:12979:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000400:00000200:14.0:1547251579.741626:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.1.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:6.0:1547251579.741626:0:13355:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251579.741627:0:12979:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:14.0:1547251579.741629:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:6.0:1547251579.741629:0:13355:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251579.741629:0:12979:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251579.741630:0:12981:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:5.0:1547251579.741631:0:12981:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:5.0:1547251579.741633:0:12981:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:14.0:1547251579.741634:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251579.741635:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:5.0:1547251579.741635:0:12981:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:14.0:1547251579.741636:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:14.0:1547251579.741639:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:12.2:1547251579.741640:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000100:00000200:14.0:1547251579.741643:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:12.2:1547251579.741643:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:0.2:1547251579.741646:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:0.2:1547251579.741648:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:4.0:1547251579.741649:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251579.741651:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251579.741652:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251579.741652:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000400:00000200:4.0:1547251579.741654:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990220 00000400:00000200:5.0:1547251579.741655:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00000400:00000200:4.0:1547251579.741655:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251579.741656:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251579.741658:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:5.0:1547251579.741659:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251579.741661:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251579.741662:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251579.741663:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:5.0:1547251579.741664:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251579.741664:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:5.0:1547251579.741665:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:4.0:1547251579.741665:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b198 00000400:00000200:5.0:1547251579.741666:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad660 00000400:00000200:4.0:1547251579.741666:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251579.741667:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251579.741669:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:5.0:1547251579.741671:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251579.741671:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251579.741673:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:5.0:1547251579.741674:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251579.741675:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:5.0:1547251579.741676:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251579.741677:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:9.2:1547251579.775701:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.2:1547251579.775717:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:12.0:1547251579.775754:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251579.775758:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251579.775762:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251579.775766:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510260 00000400:00000200:12.0:1547251579.775769:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 28352 00000100:00000200:12.0:1547251579.775772:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:14.0:1547251579.775774:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:14.0:1547251579.775778:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251579.775782:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251579.775782:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251579.775784:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251579.775786:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000400:00000200:14.0:1547251579.775787:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510250 00000800:00000200:12.0:1547251579.775787:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251579.775788:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251579.775790:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251579.775792:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 28576 00000400:00000200:12.0:1547251579.775792:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251579.775795:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 26 MB=0x5c38cb3510240 00000100:00000200:14.0:1547251579.775796:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:12.0:1547251579.775798:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0xd4cc9 [1] + 0 00000400:00000200:12.0:1547251579.775800:0:12665:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d4c8 00000100:00000200:12.0:1547251579.775801:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:14.0:1547251579.775807:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:12.0:1547251579.775807:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d4c8 00000400:00000200:12.0:1547251579.775808:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251579.775809:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251579.775810:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251579.775811:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:14.0:1547251579.775812:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251579.775813:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:14.0:1547251579.775815:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251579.775817:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:14.0:1547251579.775818:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:14.0:1547251579.775820:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251579.775820:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251579.775824:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000100:00000200:14.0:1547251579.775827:0:13374:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821728 00000400:00000200:12.0:1547251579.775827:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510270 00000100:00000200:13.0:1547251579.775829:0:13371:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821712 00000400:00000200:12.0:1547251579.775830:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 28800 00010000:00000200:14.0:1547251579.775833:0:13374:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97bae1af1200 x1622383954821728/t0(0) o400->cache1-MDT0000-lwp-OST0009_UUID@10.0.11.93@o2ib10:640/0 lens 224/224 e 0 to 0 dl 1547251585 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:12.0:1547251579.775833:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00010000:00000200:13.0:1547251579.775837:0:13371:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b0a8cac200 x1622383954821712/t0(0) o400->cache1-MDT0000-lwp-OST000a_UUID@10.0.11.93@o2ib10:640/0 lens 224/224 e 0 to 0 dl 1547251585 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:12.0:1547251579.775837:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251579.775838:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251579.775840:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:14.0:1547251579.775842:0:13374:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821728, offset 192 00000800:00000200:12.0:1547251579.775842:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000100:00000200:15.0:1547251579.775844:0:12987:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821744 00000400:00000200:14.0:1547251579.775845:0:13374:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000100:00000200:13.0:1547251579.775847:0:13371:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821712, offset 192 00010000:00000200:15.0:1547251579.775849:0:12987:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97bae1af2a00 x1622383954821744/t0(0) o400->cache1-MDT0000-lwp-OST000b_UUID@10.0.11.93@o2ib10:640/0 lens 224/224 e 0 to 0 dl 1547251585 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:14.0:1547251579.775850:0:13374:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:13.0:1547251579.775851:0:13371:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000400:00000200:14.0:1547251579.775853:0:13374:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251579.775855:0:13374:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:15.0:1547251579.775857:0:12987:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821744, offset 192 00000400:00000200:14.0:1547251579.775857:0:13374:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9730, 9731] 00000400:00000200:14.0:1547251579.775859:0:13374:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251579.775861:0:12987:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000400:00000200:14.0:1547251579.775866:0:13374:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000100:00000200:0.0:1547251579.775868:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821696 00000800:00000200:14.0:1547251579.775870:0:13374:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:13.0:1547251579.775871:0:13371:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00010000:00000200:0.0:1547251579.775874:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2dfc50 x1622383954821696/t0(0) o400->b99b3dbb-ff3e-ba67-561d-3f0ab43da035@10.0.11.93@o2ib10:640/0 lens 224/224 e 0 to 0 dl 1547251585 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:13.0:1547251579.775875:0:13371:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251579.775876:0:13371:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251579.775878:0:13371:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9731, 9731] 00000400:00000200:13.0:1547251579.775879:0:13371:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000100:00000200:0.0:1547251579.775881:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954821696, offset 192 00000400:00000200:0.0:1547251579.775884:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:13.0:1547251579.775885:0:13371:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.1.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000400:00000200:0.0:1547251579.775887:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000800:00000200:13.0:1547251579.775888:0:13371:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:15.0:1547251579.775889:0:12987:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00000400:00000200:0.0:1547251579.775890:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251579.775891:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251579.775892:0:12987:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251579.775892:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9731, 9732] 00000400:00000200:15.0:1547251579.775894:0:12987:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251579.775894:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251579.775895:0:12987:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9731, 9732] 00000400:00000200:15.0:1547251579.775898:0:12987:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:15.0:1547251579.775904:0:12987:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.1.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251579.775908:0:12987:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:14.0:1547251579.775909:0:13374:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251579.775909:0:13371:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251579.775910:0:13371:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:14.0:1547251579.775911:0:13374:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:13.0:1547251579.775911:0:13371:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:14.0:1547251579.775912:0:13374:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:0.0:1547251579.775912:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251579.775914:0:13371:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:0.0:1547251579.775915:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:15.0:1547251579.775916:0:12987:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251579.775916:0:13374:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:15.0:1547251579.775917:0:12987:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (21)++ 00000800:00000200:15.0:1547251579.775918:0:12987:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (22)++ 00000800:00000200:0.0:1547251579.775918:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251579.775919:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:0.0:1547251579.775920:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:15.0:1547251579.775921:0:12987:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (23)-- 00000800:00000200:0.0:1547251579.775923:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:7.2:1547251579.775927:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (22)++ 00000100:00000200:0.0:1547251579.775927:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:9.2:1547251579.775931:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:14.0:1547251579.775947:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (23)++ 00000800:00000200:14.0:1547251579.775949:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (24)-- 00000400:00000200:14.0:1547251579.775951:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b2a8 00000400:00000200:14.0:1547251579.775952:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:12.0:1547251579.775953:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (23)++ 00000400:00000200:14.0:1547251579.775954:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251579.775955:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (24)-- 00000800:00000200:14.0:1547251579.775956:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (23)-- 00000800:00000200:14.0:1547251579.775957:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000400:00000200:12.0:1547251579.775957:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000800:00000200:14.0:1547251579.775958:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000400:00000200:12.0:1547251579.775958:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251579.775959:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1198 00000400:00000200:14.0:1547251579.775960:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251579.775960:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:14.0:1547251579.775962:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251579.775962:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251579.775963:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:14.0:1547251579.775964:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251579.775964:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:14.0:1547251579.775965:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:14.0:1547251579.775966:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:12.0:1547251579.775966:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00000400:00000200:12.0:1547251579.775966:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251579.775968:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251579.775970:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:7.2:1547251579.807424:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:5.2:1547251579.807440:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:10.0:1547251579.807482:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251579.807486:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251579.807486:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251579.807489:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251579.807492:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251579.807494:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251579.807497:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb35102a0 00000400:00000200:11.0:1547251579.807498:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510290 00000400:00000200:10.0:1547251579.807501:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 12096 00000400:00000200:11.0:1547251579.807505:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 12320 00000100:00000200:10.0:1547251579.807508:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:10.0:1547251579.807517:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251579.807518:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:10.0:1547251579.807520:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251579.807522:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000400:00000200:11.0:1547251579.807523:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251579.807523:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251579.807525:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:10.0:1547251579.807525:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251579.807526:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251579.807527:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251579.807528:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251579.807529:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000400:00000200:10.0:1547251579.807531:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251579.807532:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251579.807535:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 26 MB=0x5c38cb3510270 00000400:00000200:11.0:1547251579.807536:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251579.807538:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0xd4cd9 [1] + 0 00000400:00000200:10.0:1547251579.807541:0:12664:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d5d8 00000100:00000200:10.0:1547251579.807542:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:10.0:1547251579.807549:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d5d8 00000400:00000200:10.0:1547251579.807550:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251579.807551:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251579.807553:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251579.807554:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251579.807556:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:10.0:1547251579.807557:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000100:00000200:11.0:1547251579.807561:0:12983:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821792 00010000:00000200:11.0:1547251579.807567:0:12983:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b17b07b300 x1622383954821792/t0(0) o400->cache1-MDT0000-lwp-OST0004_UUID@10.0.11.91@o2ib10:640/0 lens 224/224 e 0 to 0 dl 1547251585 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:8.0:1547251579.807568:0:13370:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821776 00000100:00000200:11.0:1547251579.807576:0:12983:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821792, offset 192 00010000:00000200:8.0:1547251579.807576:0:13370:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ba960ee300 x1622383954821776/t0(0) o400->cache1-MDT0000-lwp-OST0003_UUID@10.0.11.91@o2ib10:640/0 lens 224/224 e 0 to 0 dl 1547251585 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:11.0:1547251579.807580:0:12983:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000100:00000200:8.0:1547251579.807603:0:13370:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821776, offset 192 00000400:00000200:8.0:1547251579.807607:0:13370:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000100:00000200:0.0:1547251579.807609:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821744 00010000:00000200:0.0:1547251579.807615:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2ddc50 x1622383954821744/t0(0) o400->dc7951f7-3e71-e9c4-2e5e-5f95f4837d68@10.0.11.91@o2ib10:640/0 lens 224/224 e 0 to 0 dl 1547251585 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:0.0:1547251579.807621:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954821744, offset 192 00000400:00000200:0.0:1547251579.807625:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:11.0:1547251579.807626:0:12983:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:11.0:1547251579.807629:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251579.807629:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:11.0:1547251579.807630:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251579.807631:0:12983:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9737, 9736] 00000400:00000200:0.0:1547251579.807631:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251579.807633:0:12983:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:0.0:1547251579.807633:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251579.807634:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9737, 9736] 00000400:00000200:0.0:1547251579.807636:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251579.807639:0:12983:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251579.807641:0:12983:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251579.807643:0:13370:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251579.807647:0:13370:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251579.807648:0:13370:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251579.807649:0:13370:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9737, 9738] 00000400:00000200:8.0:1547251579.807651:0:13370:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:8.0:1547251579.807657:0:13370:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.91@o2ib10(10.1.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251579.807659:0:13370:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:0.0:1547251579.807663:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251579.807666:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:11.0:1547251579.807667:0:12983:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251579.807667:0:13370:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251579.807668:0:12983:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:8.0:1547251579.807668:0:13370:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251579.807669:0:12983:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:8.0:1547251579.807669:0:13370:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (22)++ 00000800:00000200:0.0:1547251579.807669:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251579.807670:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:11.0:1547251579.807671:0:12983:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:8.0:1547251579.807672:0:13370:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:0.0:1547251579.807672:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:0.0:1547251579.807674:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (23)-- 00000100:00000200:0.0:1547251579.807678:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000400:00000200:11.0:1547251579.807680:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510280 00000400:00000200:11.0:1547251579.807683:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 12544 00000800:00000200:7.2:1547251579.807683:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (22)++ 00000100:00000200:11.0:1547251579.807685:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:5.2:1547251579.807688:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (22)++ 00000400:00000200:11.0:1547251579.807692:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251579.807695:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (23)++ 00000800:00000200:11.0:1547251579.807697:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (24)-- 00000800:00000200:11.0:1547251579.807698:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251579.807700:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:11.0:1547251579.807702:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (24)-- 00000800:00000200:10.0:1547251579.807702:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000400:00000200:11.0:1547251579.807704:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bdd0 00000400:00000200:11.0:1547251579.807705:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251579.807705:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000400:00000200:10.0:1547251579.807707:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00000400:00000200:10.0:1547251579.807708:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251579.807710:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000100:00000200:8.0:1547251579.807710:0:13370:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821760 00000400:00000200:10.0:1547251579.807711:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251579.807712:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251579.807713:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251579.807714:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00010000:00000200:8.0:1547251579.807714:0:13370:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ba960ebf00 x1622383954821760/t0(0) o400->cache1-MDT0000-lwp-OST0005_UUID@10.0.11.91@o2ib10:640/0 lens 224/224 e 0 to 0 dl 1547251585 ref 1 fl Interpret:H/0/0 rc 0/0 00000800:00000200:11.0:1547251579.807715:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251579.807715:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:10.0:1547251579.807716:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:11.0:1547251579.807717:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665220 00000400:00000200:11.0:1547251579.807718:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:8.0:1547251579.807719:0:13370:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821760, offset 192 00000400:00000200:11.0:1547251579.807721:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000400:00000200:8.0:1547251579.807722:0:13370:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000800:00000200:11.0:1547251579.807723:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000400:00000200:8.0:1547251579.807725:0:13370:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251579.807727:0:13370:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251579.807728:0:13370:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251579.807729:0:13370:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9738, 9738] 00000400:00000200:8.0:1547251579.807731:0:13370:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:8.0:1547251579.807736:0:13370:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.1.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251579.807738:0:13370:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:8.0:1547251579.807740:0:13370:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251579.807741:0:13370:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:8.0:1547251579.807742:0:13370:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:8.0:1547251579.807744:0:13370:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000800:00000200:5.2:1547251579.807755:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251579.807767:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251579.807773:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000400:00000200:10.0:1547251579.807775:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6883b8 00000400:00000200:10.0:1547251579.807777:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251579.807780:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251579.807780:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251579.807782:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:14.0:1547251580.026705:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443712, portal 4 00000100:00000200:14.0:1547251580.026709:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443712, offset 0 00000100:00000200:10.0:1547251580.026709:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443680, portal 4 00000400:00000200:14.0:1547251580.026712:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000100:00000200:10.0:1547251580.026713:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443680, offset 0 00000400:00000200:14.0:1547251580.026715:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:14.0:1547251580.026717:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23007] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:10.0:1547251580.026717:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:14.0:1547251580.026720:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23003] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23007] 00000400:00000200:10.0:1547251580.026720:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:10.0:1547251580.026722:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23007] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:14.0:1547251580.026723:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:14.0:1547251580.026724:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251580.026725:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251580.026725:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23004] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23007] 00000400:00000200:14.0:1547251580.026727:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9732, 9733] 00000400:00000200:14.0:1547251580.026728:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:10.0:1547251580.026728:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:10.0:1547251580.026729:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251580.026731:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251580.026732:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9733, 9733] 00000100:00000200:2.0:1547251580.026732:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443696, portal 4 00000400:00000200:14.0:1547251580.026733:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:10.0:1547251580.026734:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000800:00000200:14.0:1547251580.026736:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000100:00000200:2.0:1547251580.026736:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443696, offset 0 00000400:00000200:10.0:1547251580.026739:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000400:00000200:2.0:1547251580.026740:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:10.0:1547251580.026741:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:14.0:1547251580.026743:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251580.026744:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:10.0:1547251580.026744:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251580.026745:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:2.0:1547251580.026745:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000800:00000200:10.0:1547251580.026746:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:10.0:1547251580.026747:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:2.0:1547251580.026747:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23007] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:14.0:1547251580.026748:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:10.0:1547251580.026750:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:2.0:1547251580.026751:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:254, d:10, s:23005] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23007] 00000400:00000200:2.0:1547251580.026753:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:2.0:1547251580.026754:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251580.026756:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251580.026757:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9733, 9734] 00000400:00000200:2.0:1547251580.026759:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000800:00000200:9.2:1547251580.026762:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:2.0:1547251580.026763:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251580.026766:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:9.2:1547251580.026769:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:2.0:1547251580.026769:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251580.026770:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:2.0:1547251580.026771:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:2.0:1547251580.026774:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:14.0:1547251580.026777:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:14.0:1547251580.026779:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000100:00000200:14.0:1547251580.026781:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a5100 x1622384012443712/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251587 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251580.026781:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251580.026784:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000400:00000200:14.0:1547251580.026786:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00000400:00000200:14.0:1547251580.026787:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:12.0:1547251580.026788:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824753300 x1622384012443680/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251587 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251580.026790:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251580.026791:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:14.0:1547251580.026793:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000400:00000200:12.0:1547251580.026793:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914198 00000800:00000200:14.0:1547251580.026794:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:12.0:1547251580.026794:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251580.026797:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251580.026799:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:7.2:1547251580.026823:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:14.0:1547251580.026838:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:14.0:1547251580.026845:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:14.0:1547251580.026847:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b6000 x1622384012443696/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251587 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251580.026851:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:14.0:1547251580.026852:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06aa18 00000400:00000200:14.0:1547251580.026852:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251580.026854:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251580.026856:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:9.2:1547251580.026912:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:7.2:1547251580.026917:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:12.0:1547251580.026923:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251580.026926:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.1.11.93@o2ib10 00000800:00000200:14.0:1547251580.026929:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000400:00000200:12.0:1547251580.026929:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:14.0:1547251580.026931:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251580.026934:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04040 00000400:00000200:14.0:1547251580.026935:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251580.026937:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f94d [1] + 192 00000400:00000200:14.0:1547251580.026945:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04020 00000400:00000200:14.0:1547251580.026948:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f95d [1] + 192 00000100:00000200:12.0:1547251580.026953:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a5100 x1622384012443712/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251587 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251580.026964:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:14.0:1547251580.026966:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824753300 x1622384012443680/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251587 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251580.026966:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251580.026967:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251580.026968:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251580.026970:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251580.026972:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251580.026974:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251580.026977:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04030 00000400:00000200:14.0:1547251580.026978:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:15.0:1547251580.026981:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a5100 x1622384012443712/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251587 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251580.026981:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000800:00000200:14.0:1547251580.026982:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:14.0:1547251580.026984:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:14.0:1547251580.026986:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:14.0:1547251580.026987:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:15.0:1547251580.026988:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a5100 x1622384012443712/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251587 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251580.026994:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:12.0:1547251580.026998:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f97d [1] + 192 00010000:00080000:15.0:1547251580.027001:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a5100 x1622384012443712/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251587 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251580.027002:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824753300 x1622384012443680/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251587 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:10.0:1547251580.027008:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824753300 x1622384012443680/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251587 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251580.027013:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a9144c8 00000100:00000200:12.0:1547251580.027015:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b6000 x1622384012443696/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251587 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:10.0:1547251580.027017:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824753300 x1622384012443680/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251587 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:12.0:1547251580.027023:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251580.027024:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251580.027025:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251580.027026:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000100:00000200:2.0:1547251580.027033:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b6000 x1622384012443696/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251587 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251580.027038:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b6000 x1622384012443696/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251587 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251580.027043:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00010000:00080000:2.0:1547251580.027046:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b6000 x1622384012443696/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251587 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251581.434732:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012443744, portal 10 00000100:00000200:5.0:1547251581.434733:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012443760, portal 10 00000100:00000200:5.0:1547251581.434736:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012443760, offset 0 00000100:00000200:4.0:1547251581.434736:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012443744, offset 0 00000400:00000200:5.0:1547251581.434739:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251581.434743:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251581.434743:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251581.434746:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251581.434747:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c04070 00000400:00000200:4.0:1547251581.434749:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c04060 00000400:00000200:5.0:1547251581.434750:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 58912 00000400:00000200:4.0:1547251581.434753:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 59136 00000100:00000200:5.0:1547251581.434758:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:7.0:1547251581.434759:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012443728, portal 25 00000100:00000200:6.0:1547251581.434759:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012443776, portal 10 00000400:00000200:5.0:1547251581.434762:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251581.434762:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:7.0:1547251581.434766:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 26, xid 1622384012443728, offset 0 00000100:00000200:6.0:1547251581.434766:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012443776, offset 0 00000400:00000200:4.0:1547251581.434766:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251581.434767:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c433000 x1622384012443760/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251581.434774:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad4c8 00000400:00000200:5.0:1547251581.434775:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:4.0:1547251581.434776:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c434200 x1622384012443744/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251581.434777:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251581.434782:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b6e8 00000400:00000200:4.0:1547251581.434783:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251581.434785:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251581.434785:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251581.434787:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251581.434789:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251581.434790:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251581.434793:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 26 MB=0x5c38cb6c04050 00000400:00000200:6.0:1547251581.434793:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c04080 00000400:00000200:7.0:1547251581.434796:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-0@lo of length 224/224 into md 0xd4d19 [1] + 0 00000400:00000200:6.0:1547251581.434796:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 59360 00000100:00000200:5.0:1547251581.434798:0:12979:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443760 00000400:00000200:7.0:1547251581.434800:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d000 00000100:00000200:6.0:1547251581.434800:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:4.0:1547251581.434800:0:12981:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443744 00000100:00000200:7.0:1547251581.434802:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00010000:00000200:5.0:1547251581.434803:0:12979:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae5c436600 x1622384012443760/t0(0) o400->cache1-MDT0000-lwp-OST0000_UUID@0@lo:642/0 lens 224/224 e 0 to 0 dl 1547251587 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:4.0:1547251581.434803:0:12981:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444b900 x1622384012443744/t0(0) o400->cache1-MDT0000-lwp-MDT0000_UUID@0@lo:642/0 lens 224/224 e 0 to 0 dl 1547251587 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:6.0:1547251581.434807:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251581.434810:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c434e00 x1622384012443776/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251581.434812:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d000 00000400:00000200:7.0:1547251581.434814:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251581.434817:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990cc0 00000400:00000200:6.0:1547251581.434818:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:5.0:1547251581.434819:0:12979:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012443760, offset 192 00000100:00000200:4.0:1547251581.434819:0:12981:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012443744, offset 192 00000100:00000200:7.0:1547251581.434820:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c436000 x1622384012443728/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251588 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251581.434820:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251581.434826:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:7.0:1547251581.434827:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251581.434828:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:5.0:1547251581.434828:0:12979:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251581.434831:0:12979:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251581.434831:0:12981:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:7.0:1547251581.434834:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012443792, portal 10 00000400:00000200:5.0:1547251581.434834:0:12979:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c04070 00000400:00000200:4.0:1547251581.434834:0:12981:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:7.0:1547251581.434836:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012443792, offset 0 00000400:00000200:5.0:1547251581.434837:0:12979:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaeb81 [1] + 192 00000400:00000200:4.0:1547251581.434837:0:12981:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c04060 00000400:00000200:7.0:1547251581.434839:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:6.0:1547251581.434839:0:13355:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443776 00000400:00000200:4.0:1547251581.434840:0:12981:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaeb71 [1] + 192 00000400:00000200:7.0:1547251581.434842:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:5.0:1547251581.434843:0:12979:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c433000 x1622384012443760/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251581.434845:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c04090 00010000:00000200:6.0:1547251581.434845:0:13355:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a765df3000 x1622384012443776/t0(0) o400->cache1-MDT0000-lwp-OST0002_UUID@0@lo:642/0 lens 224/224 e 0 to 0 dl 1547251587 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:0.0:1547251581.434845:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443728 00000400:00000200:7.0:1547251581.434848:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 59584 00000400:00000200:5.0:1547251581.434850:0:12979:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251581.434851:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:4.0:1547251581.434851:0:12981:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c434200 x1622384012443744/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00010000:00000200:0.0:1547251581.434851:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2d9450 x1622384012443728/t0(0) o400->a65be517-33e7-fa36-0de3-da9e02df4f8d@0@lo:642/0 lens 224/224 e 0 to 0 dl 1547251587 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:7.0:1547251581.434853:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251581.434854:0:12979:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad4c8 00000100:00000200:6.0:1547251581.434855:0:13355:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012443776, offset 192 00000400:00000200:5.0:1547251581.434855:0:12979:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:7.0:1547251581.434856:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c431800 x1622384012443792/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251581.434856:0:12979:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251581.434857:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:0.0:1547251581.434858:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384012443728, offset 192 00000400:00000200:0.0:1547251581.434860:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251581.434861:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea4c8 00000400:00000200:7.0:1547251581.434862:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251581.434863:0:13355:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:0.0:1547251581.434863:0:12929:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251581.434864:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251581.434864:0:12981:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b6e8 00000400:00000200:4.0:1547251581.434865:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251581.434866:0:13355:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:0.0:1547251581.434866:0:12929:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 25 MB=0x5c38cb6c04050 00000100:00000200:5.0:1547251581.434867:0:12979:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443792 00000400:00000200:4.0:1547251581.434867:0:12981:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251581.434869:0:13355:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c04080 00000400:00000200:0.0:1547251581.434869:0:12929:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 19 from 12345-0@lo of length 192/192 into md 0xaeb91 [1] + 192 00010000:00000200:5.0:1547251581.434870:0:12979:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae37930600 x1622384012443792/t0(0) o400->cache1-MDT0000-lwp-OST0001_UUID@0@lo:642/0 lens 224/224 e 0 to 0 dl 1547251587 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:6.0:1547251581.434872:0:13355:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaeba1 [1] + 192 00000100:00000200:5.0:1547251581.434876:0:12979:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012443792, offset 192 00000100:00000200:0.0:1547251581.434876:0:12929:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c436000 x1622384012443728/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251581.434879:0:12979:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251581.434882:0:12979:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:0.0:1547251581.434884:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251581.434885:0:12979:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c04090 00000100:00000200:6.0:1547251581.434887:0:13355:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c434e00 x1622384012443776/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251581.434894:0:13355:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251581.434895:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c434200 x1622384012443744/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251581.434896:0:13355:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990cc0 00000400:00000200:6.0:1547251581.434897:0:13355:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251581.434898:0:13355:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251581.434900:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c434200 x1622384012443744/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251581.434905:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b198 00000400:00000200:0.0:1547251581.434907:0:12929:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665b28 00000400:00000200:0.0:1547251581.434907:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251581.434909:0:12979:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaebb1 [1] + 192 00000400:00000200:0.0:1547251581.434909:0:12929:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251581.434912:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c436000 x1622384012443728/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251581.434913:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000100:00000200:7.0:1547251581.434918:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c436000 x1622384012443728/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251581.434922:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00000100:00000200:6.0:1547251581.434924:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c434e00 x1622384012443776/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251581.434931:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c434e00 x1622384012443776/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251581.434936:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990880 00000100:00000200:5.0:1547251581.434947:0:12979:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c431800 x1622384012443792/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251581.434956:0:12979:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251581.434958:0:12979:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad4c8 00000400:00000200:5.0:1547251581.434959:0:12979:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251581.434960:0:12979:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251581.434968:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c433000 x1622384012443760/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251581.434972:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c433000 x1622384012443760/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251581.434976:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad660 00000100:00000200:7.0:1547251581.434978:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c431800 x1622384012443792/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251581.434982:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c431800 x1622384012443792/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251588 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251581.434986:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000100:00000200:1.0:1547251582.138686:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443824, portal 4 00000100:00000200:1.0:1547251582.138688:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443824, offset 0 00000400:00000200:1.0:1547251582.138692:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:1.0:1547251582.138694:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:1.0:1547251582.138696:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23008] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251582.138700:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23005] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23008] 00000400:00000200:1.0:1547251582.138702:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:1.0:1547251582.138703:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251582.138706:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251582.138707:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9738, 9739] 00000100:00000200:11.0:1547251582.138709:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443808, portal 4 00000400:00000200:1.0:1547251582.138709:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:1.0:1547251582.138713:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000100:00000200:11.0:1547251582.138714:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443808, offset 0 00000800:00000200:1.0:1547251582.138716:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:11.0:1547251582.138719:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:1.0:1547251582.138719:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251582.138720:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:1.0:1547251582.138721:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000100:00000200:12.0:1547251582.138722:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443840, portal 4 00000400:00000200:11.0:1547251582.138722:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:11.0:1547251582.138724:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23008] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:1.0:1547251582.138724:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000100:00000200:12.0:1547251582.138725:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443840, offset 0 00000400:00000200:12.0:1547251582.138728:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:11.0:1547251582.138728:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23006] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23008] 00000400:00000200:11.0:1547251582.138730:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:12.0:1547251582.138731:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:11.0:1547251582.138732:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251582.138733:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23009] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:11.0:1547251582.138733:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251582.138734:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9739, 9739] 00000400:00000200:11.0:1547251582.138736:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:12.0:1547251582.138737:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23006] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23009] 00000400:00000200:12.0:1547251582.138739:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:12.0:1547251582.138741:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251582.138741:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000400:00000200:12.0:1547251582.138742:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251582.138744:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9739, 9740] 00000800:00000200:11.0:1547251582.138744:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:12.0:1547251582.138745:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251582.138746:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251582.138748:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:11.0:1547251582.138749:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000400:00000200:12.0:1547251582.138750:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251582.138752:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000800:00000200:12.0:1547251582.138754:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:12.0:1547251582.138757:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251582.138758:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:12.0:1547251582.138759:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000800:00000200:12.0:1547251582.138762:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:7.2:1547251582.138773:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251582.138789:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251582.138797:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000100:00000200:11.0:1547251582.138802:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b6300 x1622384012443824/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.2:1547251582.138802:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:5.2:1547251582.138806:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251582.138807:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:11.0:1547251582.138809:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2110 00000400:00000200:11.0:1547251582.138810:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251582.138810:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251582.138812:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000400:00000200:11.0:1547251582.138814:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000100:00000200:10.0:1547251582.138815:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824753900 x1622384012443808/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251582.138816:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:11.0:1547251582.138818:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251582.138820:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000400:00000200:10.0:1547251582.138821:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b550 00000400:00000200:10.0:1547251582.138822:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:11.0:1547251582.138823:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f0f00 x1622384012443840/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251582.138825:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251582.138828:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038ee0 00000800:00000200:10.0:1547251582.138828:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:11.0:1547251582.138829:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251582.138829:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:10.0:1547251582.138831:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:11.0:1547251582.138832:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251582.138834:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:5.2:1547251582.138900:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251582.138906:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:7.2:1547251582.138912:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:10.0:1547251582.138915:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251582.138918:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251582.138919:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251582.138923:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251582.138923:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251582.138927:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251582.138928:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c040b0 00000400:00000200:11.0:1547251582.138930:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c040c0 00000400:00000200:10.0:1547251582.138932:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd69a9 [1] + 192 00000400:00000200:11.0:1547251582.138936:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd69d9 [1] + 192 00000100:00000200:10.0:1547251582.138948:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b6300 x1622384012443824/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251582.138960:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251582.138961:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f0f00 x1622384012443840/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251582.138962:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:10.0:1547251582.138964:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251582.138965:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251582.138967:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251582.138969:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251582.138972:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251582.138973:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000400:00000200:10.0:1547251582.138973:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251582.138975:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251582.138976:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000100:00000200:1.0:1547251582.138976:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b6300 x1622384012443824/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251582.138977:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c040a0 00000800:00000200:11.0:1547251582.138978:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:11.0:1547251582.138980:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:11.0:1547251582.138981:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000100:00000200:1.0:1547251582.138983:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b6300 x1622384012443824/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251582.138989:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00000400:00000200:10.0:1547251582.138993:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd69b9 [1] + 192 00010000:00080000:1.0:1547251582.138993:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b6300 x1622384012443824/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:12.0:1547251582.138997:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f0f00 x1622384012443840/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251582.139002:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f0f00 x1622384012443840/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251582.139007:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038cc0 00000100:00000200:10.0:1547251582.139009:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824753900 x1622384012443808/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:12.0:1547251582.139010:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f0f00 x1622384012443840/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251582.139018:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251582.139019:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251582.139021:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251582.139023:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:11.0:1547251582.139028:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824753900 x1622384012443808/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251582.139035:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824753900 x1622384012443808/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251582.139040:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bdd0 00010000:00080000:11.0:1547251582.139045:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824753900 x1622384012443808/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251582.602681:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443856, portal 4 00000100:00000200:8.0:1547251582.602685:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443856, offset 0 00000400:00000200:8.0:1547251582.602688:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251582.602690:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000100:00000200:1.0:1547251582.602692:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443872, portal 4 00000400:00000200:8.0:1547251582.602693:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23010] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:1.0:1547251582.602695:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443872, offset 0 00000400:00000200:8.0:1547251582.602697:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23006] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23010] 00000400:00000200:1.0:1547251582.602698:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251582.602699:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000100:00000200:13.0:1547251582.602700:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443888, portal 4 00000400:00000200:8.0:1547251582.602701:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251582.602701:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251582.602702:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:13.0:1547251582.602703:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443888, offset 0 00000400:00000200:1.0:1547251582.602703:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23010] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251582.602704:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9732, 9732] 00000400:00000200:13.0:1547251582.602706:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251582.602706:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:1.0:1547251582.602707:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23007] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23010] 00000400:00000200:13.0:1547251582.602709:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:1.0:1547251582.602709:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:8.0:1547251582.602710:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:1.0:1547251582.602711:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251582.602712:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23010] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:8.0:1547251582.602713:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:1.0:1547251582.602713:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251582.602715:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23008] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23010] 00000400:00000200:1.0:1547251582.602715:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9732, 9733] 00000800:00000200:8.0:1547251582.602716:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:1.0:1547251582.602716:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:8.0:1547251582.602717:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000400:00000200:13.0:1547251582.602718:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:8.0:1547251582.602718:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000400:00000200:13.0:1547251582.602719:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251582.602721:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251582.602721:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:1.0:1547251582.602721:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:13.0:1547251582.602722:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9733, 9733] 00000400:00000200:13.0:1547251582.602724:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:1.0:1547251582.602724:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:1.0:1547251582.602728:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:13.0:1547251582.602729:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:1.0:1547251582.602731:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:13.0:1547251582.602732:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:1.0:1547251582.602732:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:13.0:1547251582.602735:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251582.602736:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:1.0:1547251582.602736:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:13.0:1547251582.602737:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:0.2:1547251582.602739:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:13.0:1547251582.602740:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:0.2:1547251582.602747:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:12.2:1547251582.602758:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251582.602796:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:5.0:1547251582.602800:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:4.0:1547251582.602800:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251582.602802:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:5.0:1547251582.602804:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824752700 x1622384012443856/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251582.602805:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b6600 x1622384012443872/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251582.602812:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6883b8 00000400:00000200:4.0:1547251582.602812:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2660 00000400:00000200:5.0:1547251582.602814:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251582.602814:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251582.602817:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251582.602817:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251582.602820:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251582.602821:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251582.602822:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251582.602823:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251582.602824:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251582.602825:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:5.0:1547251582.602826:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:4.0:1547251582.602827:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af4b00 x1622384012443888/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251582.602833:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b990 00000400:00000200:4.0:1547251582.602834:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251582.602837:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251582.602839:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:0.2:1547251582.602903:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:12.2:1547251582.602908:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:0.2:1547251582.602913:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:5.0:1547251582.602916:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251582.602921:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000800:00000200:4.0:1547251582.602922:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251582.602924:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251582.602926:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251582.602929:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251582.602931:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c040f0 00000400:00000200:4.0:1547251582.602932:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c040d0 00000400:00000200:5.0:1547251582.602935:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3775 [1] + 192 00000400:00000200:4.0:1547251582.602950:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3755 [1] + 192 00000100:00000200:5.0:1547251582.602955:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af4b00 x1622384012443888/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251582.602973:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251582.602975:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824752700 x1622384012443856/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251582.602976:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:5.0:1547251582.602978:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:5.0:1547251582.602979:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251582.602981:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251582.602983:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251582.602986:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251582.602987:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000100:00000200:13.0:1547251582.602988:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af4b00 x1622384012443888/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251582.602988:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251582.602990:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000400:00000200:5.0:1547251582.602991:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c040e0 00000800:00000200:4.0:1547251582.602991:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251582.602993:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000100:00000200:13.0:1547251582.602994:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af4b00 x1622384012443888/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251582.602995:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:4.0:1547251582.602996:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:13.0:1547251582.602999:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b3b8 00000400:00000200:5.0:1547251582.603002:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3765 [1] + 192 00010000:00080000:13.0:1547251582.603003:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af4b00 x1622384012443888/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251582.603007:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b6600 x1622384012443872/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251582.603017:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251582.603018:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251582.603020:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251582.603021:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:1.0:1547251582.603028:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b6600 x1622384012443872/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:1.0:1547251582.603033:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b6600 x1622384012443872/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251582.603037:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00010000:00080000:1.0:1547251582.603041:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b6600 x1622384012443872/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251582.603049:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824752700 x1622384012443856/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251582.603055:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824752700 x1622384012443856/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251582.603059:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688e58 00010000:00080000:8.0:1547251582.603063:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824752700 x1622384012443856/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251589 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251583.002675:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443904, portal 4 00000100:00000200:8.0:1547251583.002678:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443904, offset 0 00000400:00000200:8.0:1547251583.002680:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251583.002683:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251583.002686:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04100 00000400:00000200:8.0:1547251583.002688:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd42f9 [1] + 0 00000400:00000200:8.0:1547251583.002690:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d688a18 00000100:00000200:8.0:1547251583.002692:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251583.002700:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688a18 00000400:00000200:8.0:1547251583.002702:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251583.002705:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824752700 x1622384012443904/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251590 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251583.002719:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00000400:00000200:8.0:1547251583.002720:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:6.0:1547251583.002720:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443920, portal 4 00000100:00000200:9.0:1547251583.002722:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443904 00000400:00000200:8.0:1547251583.002722:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251583.002723:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443920, offset 0 00000400:00000200:6.0:1547251583.002727:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251583.002730:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00010000:00000200:9.0:1547251583.002732:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3dcc50 x1622384012443904/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:644/0 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:6.0:1547251583.002732:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04110 00000400:00000200:6.0:1547251583.002735:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1f05 [1] + 0 00000400:00000200:6.0:1547251583.002737:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040b000 00000100:00000200:6.0:1547251583.002739:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:9.0:1547251583.002740:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443904, offset 192 00000400:00000200:9.0:1547251583.002743:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251583.002746:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251583.002746:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b000 00000400:00000200:6.0:1547251583.002747:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251583.002748:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04100 00000400:00000200:9.0:1547251583.002750:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaebf1 [1] + 192 00000100:00000200:6.0:1547251583.002750:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37933c00 x1622384012443920/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251590 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251583.002753:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824752700 x1622384012443904/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251590 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251583.002755:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990cc0 00000400:00000200:6.0:1547251583.002755:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251583.002756:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:9.0:1547251583.002761:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251583.002763:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ead48 00000400:00000200:9.0:1547251583.002764:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251583.002767:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251583.002770:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824752700 x1622384012443904/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251590 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251583.002773:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:8.0:1547251583.002776:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824752700 x1622384012443904/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251590 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251583.002776:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443920 00000400:00000200:8.0:1547251583.002782:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688e58 00010000:00000200:5.0:1547251583.002785:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39f450 x1622384012443920/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:644/0 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Interpret:/0/0 rc 0/0 00010000:00080000:8.0:1547251583.002786:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824752700 x1622384012443904/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251590 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251583.002792:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443920, offset 192 00000400:00000200:5.0:1547251583.002794:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251583.002798:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251583.002800:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04110 00000400:00000200:5.0:1547251583.002803:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaec01 [1] + 192 00000100:00000200:5.0:1547251583.002806:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37933c00 x1622384012443920/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251590 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251583.002817:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251583.002819:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad660 00000400:00000200:5.0:1547251583.002820:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251583.002821:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251583.002825:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37933c00 x1622384012443920/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251590 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251583.002826:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251583.002829:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37933c00 x1622384012443920/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251590 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251583.002833:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990880 00010000:00080000:6.0:1547251583.002836:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37933c00 x1622384012443920/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251590 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251583.018681:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443936, portal 4 00000100:00000200:7.0:1547251583.018684:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443936, offset 0 00000400:00000200:7.0:1547251583.018686:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251583.018689:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251583.018693:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04120 00000400:00000200:7.0:1547251583.018696:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1f35 [1] + 0 00000400:00000200:7.0:1547251583.018698:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073adf68 00000100:00000200:7.0:1547251583.018700:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:7.0:1547251583.018710:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adf68 00000400:00000200:7.0:1547251583.018711:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251583.018713:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df1500 x1622384012443936/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251590 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251583.018718:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00000400:00000200:7.0:1547251583.018718:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251583.018720:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251583.018730:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012443936 00010000:00000200:5.0:1547251583.018739:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39f050 x1622384012443936/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:644/0 lens 224/368 e 0 to 0 dl 1547251589 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251583.018746:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012443936, offset 192 00000400:00000200:5.0:1547251583.018749:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251583.018752:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251583.018755:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04120 00000400:00000200:5.0:1547251583.018757:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaec11 [1] + 192 00000100:00000200:5.0:1547251583.018761:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df1500 x1622384012443936/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251590 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251583.018770:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251583.018772:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad4c8 00000400:00000200:5.0:1547251583.018772:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251583.018774:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251583.018778:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df1500 x1622384012443936/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251590 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251583.018779:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251583.018782:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df1500 x1622384012443936/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251590 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251583.018786:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00010000:00080000:7.0:1547251583.018789:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df1500 x1622384012443936/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251590 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251585.034681:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443952, portal 4 00000100:00000200:11.0:1547251585.034684:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443952, offset 0 00000400:00000200:11.0:1547251585.034687:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:11.0:1547251585.034689:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:11.0:1547251585.034692:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23011] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:11.0:1547251585.034695:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23008] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23011] 00000400:00000200:11.0:1547251585.034697:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:11.0:1547251585.034699:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251585.034700:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251585.034701:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9734, 9734] 00000400:00000200:11.0:1547251585.034703:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:11.0:1547251585.034709:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251585.034712:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:11.0:1547251585.034715:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251585.034716:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:11.0:1547251585.034717:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:11.0:1547251585.034720:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:9.2:1547251585.034739:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000100:00000200:3.0:1547251585.034747:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443968, portal 4 00000100:00000200:3.0:1547251585.034751:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443968, offset 0 00000400:00000200:3.0:1547251585.034755:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:3.0:1547251585.034759:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:3.0:1547251585.034762:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23011] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:14.0:1547251585.034765:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000400:00000200:3.0:1547251585.034766:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23009] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23011] 00000400:00000200:3.0:1547251585.034769:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251585.034771:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:14.0:1547251585.034773:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000400:00000200:3.0:1547251585.034773:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251585.034775:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9734, 9735] 00000400:00000200:3.0:1547251585.034777:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000100:00000200:14.0:1547251585.034778:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07aa00 x1622384012443952/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251592 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251585.034780:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012443984, portal 4 00000400:00000200:14.0:1547251585.034783:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b6e8 00000100:00000200:2.0:1547251585.034783:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012443984, offset 0 00000400:00000200:14.0:1547251585.034784:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:3.0:1547251585.034784:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:14.0:1547251585.034787:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:2.0:1547251585.034787:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:3.0:1547251585.034788:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:14.0:1547251585.034789:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:2.0:1547251585.034790:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000800:00000200:14.0:1547251585.034791:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000400:00000200:2.0:1547251585.034792:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23012] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251585.034796:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23009] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23012] 00000400:00000200:2.0:1547251585.034799:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251585.034801:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251585.034802:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251585.034805:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9735, 9735] 00000400:00000200:2.0:1547251585.034807:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000800:00000200:3.0:1547251585.034811:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251585.034814:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000400:00000200:2.0:1547251585.034814:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251585.034815:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:2.0:1547251585.034818:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:3.0:1547251585.034819:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:2.0:1547251585.034821:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251585.034823:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:2.0:1547251585.034824:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:2.0:1547251585.034827:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:9.2:1547251585.034840:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:14.0:1547251585.034852:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:14.0:1547251585.034857:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:14.0:1547251585.034860:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b6f00 x1622384012443984/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251592 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251585.034864:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06aaa0 00000400:00000200:14.0:1547251585.034865:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:12.0:1547251585.034865:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:14.0:1547251585.034867:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251585.034869:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:7.2:1547251585.034871:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251585.034881:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251585.034893:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:12.0:1547251585.034897:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b6900 x1622384012443968/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251592 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251585.034899:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:12.0:1547251585.034902:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555ee0 00000400:00000200:12.0:1547251585.034903:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251585.034906:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251585.034908:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:9.2:1547251585.034933:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.2:1547251585.034938:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:7.2:1547251585.034951:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:14.0:1547251585.034954:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:14.0:1547251585.034956:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251585.034958:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000400:00000200:14.0:1547251585.034960:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251585.034961:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251585.034964:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04130 00000400:00000200:12.0:1547251585.034965:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251585.034967:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f9ad [1] + 192 00000400:00000200:12.0:1547251585.034969:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04150 00000100:00000200:14.0:1547251585.034971:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07aa00 x1622384012443952/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251592 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251585.034979:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251585.034981:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000400:00000200:12.0:1547251585.034981:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f9cd [1] + 192 00000800:00000200:14.0:1547251585.034982:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:14.0:1547251585.034983:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:12.0:1547251585.034984:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b6f00 x1622384012443984/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251592 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251585.034985:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:14.0:1547251585.034987:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251585.034990:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251585.034992:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04140 00000400:00000200:12.0:1547251585.034993:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:14.0:1547251585.034994:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f9bd [1] + 192 00000800:00000200:12.0:1547251585.034995:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000800:00000200:12.0:1547251585.034996:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000100:00000200:14.0:1547251585.034998:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b6900 x1622384012443968/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251592 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251585.034998:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251585.035000:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:12.0:1547251585.035001:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:12.0:1547251585.035002:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:14.0:1547251585.035005:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251585.035007:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:14.0:1547251585.035008:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:14.0:1547251585.035008:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000100:00000200:11.0:1547251585.035009:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07aa00 x1622384012443952/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251592 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251585.035014:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07aa00 x1622384012443952/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251592 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251585.035019:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bdd0 00000100:00000200:2.0:1547251585.035022:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b6f00 x1622384012443984/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251592 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:11.0:1547251585.035023:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b17b07aa00 x1622384012443952/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251592 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:2.0:1547251585.035030:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b6f00 x1622384012443984/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251592 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251585.035035:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00000100:00000200:3.0:1547251585.035038:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b6900 x1622384012443968/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251592 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:2.0:1547251585.035040:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b6f00 x1622384012443984/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251592 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251585.035045:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b6900 x1622384012443968/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251592 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251585.035051:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251585.035054:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b6900 x1622384012443968/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251592 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:0.0:1547251587.146681:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444000, portal 4 00000100:00000200:0.0:1547251587.146684:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444000, offset 0 00000400:00000200:0.0:1547251587.146687:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:0.0:1547251587.146689:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:0.0:1547251587.146693:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23012] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251587.146697:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23010] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23012] 00000400:00000200:0.0:1547251587.146699:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251587.146700:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251587.146702:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251587.146703:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9740, 9740] 00000400:00000200:0.0:1547251587.146705:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:0.0:1547251587.146709:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251587.146712:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:0.0:1547251587.146715:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251587.146716:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:0.0:1547251587.146717:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:0.0:1547251587.146720:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000100:00000200:8.0:1547251587.146729:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444016, portal 4 00000100:00000200:8.0:1547251587.146732:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444016, offset 0 00000400:00000200:8.0:1547251587.146737:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251587.146740:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251587.146742:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23012] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251587.146745:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23011] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23012] 00000400:00000200:8.0:1547251587.146747:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251587.146749:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251587.146750:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251587.146752:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9740, 9741] 00000400:00000200:8.0:1547251587.146754:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000100:00000200:14.0:1547251587.146755:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444032, portal 4 00000400:00000200:8.0:1547251587.146758:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000100:00000200:14.0:1547251587.146759:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444032, offset 0 00000800:00000200:8.0:1547251587.146761:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:14.0:1547251587.146763:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:8.0:1547251587.146764:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251587.146765:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000400:00000200:14.0:1547251587.146766:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:8.0:1547251587.146766:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000400:00000200:14.0:1547251587.146768:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23013] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:8.0:1547251587.146769:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)-- 00000400:00000200:14.0:1547251587.146771:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23011] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23013] 00000800:00000200:7.2:1547251587.146771:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:14.0:1547251587.146773:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:14.0:1547251587.146775:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251587.146776:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251587.146777:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9741, 9741] 00000400:00000200:14.0:1547251587.146779:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:14.0:1547251587.146784:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251587.146787:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:11.0:1547251587.146788:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:14.0:1547251587.146790:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251587.146791:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:14.0:1547251587.146792:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (24)++ 00000800:00000200:14.0:1547251587.146795:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (25)-- 00000800:00000200:11.0:1547251587.146796:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (24)-- 00000100:00000200:11.0:1547251587.146800:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328dcb00 x1622384012444000/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251587.146808:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665440 00000400:00000200:11.0:1547251587.146810:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251587.146813:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)++ 00000400:00000200:11.0:1547251587.146814:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251587.146815:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (24)-- 00000800:00000200:11.0:1547251587.146816:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:11.0:1547251587.146818:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000100:00000200:10.0:1547251587.146818:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f1200 x1622384012444032/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.2:1547251587.146820:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00000400:00000200:10.0:1547251587.146825:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:10.0:1547251587.146826:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251587.146829:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251587.146831:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:11.0:1547251587.146832:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251587.146835:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251587.146836:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000100:00000200:11.0:1547251587.146838:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960ee000 x1622384012444016/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251587.146843:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00000400:00000200:11.0:1547251587.146845:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251587.146848:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251587.146849:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:5.2:1547251587.146906:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251587.146912:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:5.2:1547251587.146918:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:10.0:1547251587.146920:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251587.146924:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251587.146925:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251587.146928:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251587.146929:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251587.146933:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251587.146934:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04170 00000400:00000200:11.0:1547251587.146937:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04180 00000400:00000200:10.0:1547251587.146946:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6a49 [1] + 192 00000400:00000200:11.0:1547251587.146950:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6a69 [1] + 192 00000100:00000200:10.0:1547251587.146955:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960ee000 x1622384012444016/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251587.146966:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251587.146968:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f1200 x1622384012444032/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251587.146968:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:10.0:1547251587.146970:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251587.146971:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251587.146973:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251587.146975:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251587.146978:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:10.0:1547251587.146979:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251587.146980:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000100:00000200:8.0:1547251587.146980:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960ee000 x1622384012444016/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251587.146981:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:11.0:1547251587.146983:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:10.0:1547251587.146983:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04160 00000800:00000200:11.0:1547251587.146984:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:11.0:1547251587.146986:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:8.0:1547251587.146986:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960ee000 x1622384012444016/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251587.146987:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:8.0:1547251587.146991:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688e58 00000400:00000200:10.0:1547251587.146994:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6a39 [1] + 192 00010000:00080000:8.0:1547251587.146995:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba960ee000 x1622384012444016/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:14.0:1547251587.147000:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f1200 x1622384012444032/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:14.0:1547251587.147005:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f1200 x1622384012444032/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251587.147010:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00000100:00000200:10.0:1547251587.147012:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328dcb00 x1622384012444000/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:14.0:1547251587.147013:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f1200 x1622384012444032/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251587.147026:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251587.147027:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251587.147028:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251587.147029:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:0.0:1547251587.147072:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328dcb00 x1622384012444000/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251587.147078:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328dcb00 x1622384012444000/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251587.147083:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665198 00010000:00080000:0.0:1547251587.147087:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328dcb00 x1622384012444000/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251587.610681:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444048, portal 4 00000100:00000200:9.0:1547251587.610684:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444048, offset 0 00000400:00000200:9.0:1547251587.610687:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251587.610690:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251587.610692:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23013] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:12.0:1547251587.610695:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444080, portal 4 00000400:00000200:9.0:1547251587.610695:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23012] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23013] 00000100:00000200:0.0:1547251587.610696:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444064, portal 4 00000100:00000200:12.0:1547251587.610698:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444080, offset 0 00000400:00000200:9.0:1547251587.610698:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000100:00000200:0.0:1547251587.610699:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444064, offset 0 00000400:00000200:9.0:1547251587.610700:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251587.610701:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251587.610702:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251587.610702:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9733, 9734] 00000400:00000200:0.0:1547251587.610702:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251587.610704:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:12.0:1547251587.610705:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:0.0:1547251587.610705:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:12.0:1547251587.610707:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23013] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251587.610707:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23013] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251587.610709:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:12.0:1547251587.610710:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23013] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23013] 00000400:00000200:0.0:1547251587.610711:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23013] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23013] 00000800:00000200:9.0:1547251587.610712:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:12.0:1547251587.610713:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:0.0:1547251587.610713:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:12.0:1547251587.610714:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251587.610715:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:0.0:1547251587.610715:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251587.610716:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251587.610716:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000400:00000200:0.0:1547251587.610716:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251587.610717:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9734, 9734] 00000800:00000200:9.0:1547251587.610717:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:0.0:1547251587.610718:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9734, 9734] 00000400:00000200:12.0:1547251587.610719:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:9.0:1547251587.610720:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:0.0:1547251587.610721:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:12.0:1547251587.610724:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251587.610726:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:12.0:1547251587.610729:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:0.2:1547251587.610730:0:12673:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:12.0:1547251587.610731:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:12.0:1547251587.610732:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:12.0:1547251587.610735:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:12.2:1547251587.610743:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000400:00000200:0.0:1547251587.610747:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251587.610749:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:0.0:1547251587.610753:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251587.610754:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (22)++ 00000800:00000200:0.0:1547251587.610755:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00000800:00000200:0.0:1547251587.610758:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (24)-- 00000800:00000200:4.0:1547251587.610786:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251587.610790:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:4.0:1547251587.610790:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251587.610792:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000100:00000200:4.0:1547251587.610794:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf5400 x1622384012444048/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251587.610796:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f1500 x1622384012444080/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251587.610802:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b0384c8 00000400:00000200:4.0:1547251587.610802:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaee0 00000400:00000200:5.0:1547251587.610804:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251587.610804:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251587.610807:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251587.610807:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251587.610810:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251587.610811:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251587.610812:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:5.0:1547251587.610814:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251587.610818:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251587.610819:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:5.0:1547251587.610821:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328dce00 x1622384012444064/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251587.610827:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665c38 00000400:00000200:5.0:1547251587.610828:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251587.610831:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251587.610832:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:0.2:1547251587.610891:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:12.2:1547251587.610896:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:12.2:1547251587.610902:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:4.0:1547251587.610905:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251587.610909:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000800:00000200:5.0:1547251587.610910:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251587.610913:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251587.610914:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251587.610918:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251587.610919:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c041b0 00000400:00000200:5.0:1547251587.610921:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04190 00000400:00000200:4.0:1547251587.610923:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe37f5 [1] + 192 00000400:00000200:5.0:1547251587.610927:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe37e5 [1] + 192 00000100:00000200:4.0:1547251587.610932:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f1500 x1622384012444080/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251587.610953:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251587.610957:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251587.610959:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000100:00000200:5.0:1547251587.610960:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf5400 x1622384012444048/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251587.610961:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251587.610963:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251587.610965:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251587.610970:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251587.610970:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000100:00000200:12.0:1547251587.610972:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f1500 x1622384012444080/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251587.610972:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251587.610974:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000400:00000200:4.0:1547251587.610974:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c041a0 00000800:00000200:5.0:1547251587.610975:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251587.610977:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000100:00000200:12.0:1547251587.610978:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f1500 x1622384012444080/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251587.610979:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:5.0:1547251587.610980:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:12.0:1547251587.610982:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038cc0 00010000:00080000:12.0:1547251587.610985:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f1500 x1622384012444080/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:4.0:1547251587.610985:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3805 [1] + 192 00000100:00000200:9.0:1547251587.610991:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf5400 x1622384012444048/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251587.610996:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf5400 x1622384012444048/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251587.611000:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea440 00000100:00000200:4.0:1547251587.611003:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328dce00 x1622384012444064/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251587.611004:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf5400 x1622384012444048/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:4.0:1547251587.611013:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251587.611014:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251587.611015:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251587.611016:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:0.0:1547251587.611022:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328dce00 x1622384012444064/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251587.611028:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328dce00 x1622384012444064/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251587.611032:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665198 00010000:00080000:0.0:1547251587.611036:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328dce00 x1622384012444064/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251594 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251588.010676:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444096, portal 4 00000100:00000200:9.0:1547251588.010679:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444096, offset 0 00000400:00000200:9.0:1547251588.010681:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251588.010684:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251588.010687:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c041c0 00000400:00000200:9.0:1547251588.010689:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4379 [1] + 0 00000400:00000200:9.0:1547251588.010691:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b8245ea990 00000100:00000200:9.0:1547251588.010693:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251588.010702:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea990 00000400:00000200:9.0:1547251588.010703:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251588.010706:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf5a00 x1622384012444096/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251595 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251588.010713:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaaa0 00000400:00000200:9.0:1547251588.010715:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251588.010716:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251588.010724:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444096 00000100:00000200:7.0:1547251588.010726:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444112, portal 4 00000100:00000200:7.0:1547251588.010729:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444112, offset 0 00000400:00000200:7.0:1547251588.010732:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00010000:00000200:8.0:1547251588.010734:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3dc850 x1622384012444096/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:649/0 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:7.0:1547251588.010735:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251588.010738:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c041d0 00000100:00000200:8.0:1547251588.010740:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444096, offset 192 00000400:00000200:7.0:1547251588.010740:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1fa5 [1] + 0 00000400:00000200:7.0:1547251588.010742:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad088 00000400:00000200:8.0:1547251588.010743:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:7.0:1547251588.010744:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251588.010745:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251588.010748:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c041c0 00000400:00000200:8.0:1547251588.010750:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaec61 [1] + 192 00000400:00000200:7.0:1547251588.010752:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad088 00000100:00000200:8.0:1547251588.010753:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf5a00 x1622384012444096/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251595 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251588.010754:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251588.010758:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df2100 x1622384012444112/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251595 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251588.010761:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251588.010764:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688e58 00000400:00000200:7.0:1547251588.010764:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00000400:00000200:8.0:1547251588.010765:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251588.010765:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251588.010767:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251588.010767:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251588.010771:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf5a00 x1622384012444096/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251595 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251588.010774:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:9.0:1547251588.010777:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf5a00 x1622384012444096/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251595 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251588.010777:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444112 00000400:00000200:9.0:1547251588.010782:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea440 00010000:00080000:9.0:1547251588.010785:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf5a00 x1622384012444096/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251595 ref 1 fl Rpc:R/0/0 rc 0/0 00010000:00000200:6.0:1547251588.010787:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39ec50 x1622384012444112/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:649/0 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251588.010794:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444112, offset 192 00000400:00000200:6.0:1547251588.010796:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251588.010799:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251588.010802:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c041d0 00000400:00000200:6.0:1547251588.010805:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaec71 [1] + 192 00000100:00000200:6.0:1547251588.010807:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df2100 x1622384012444112/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251595 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251588.010816:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251588.010819:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990880 00000400:00000200:6.0:1547251588.010820:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251588.010823:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251588.010827:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df2100 x1622384012444112/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251595 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251588.010831:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251588.010832:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df2100 x1622384012444112/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251595 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251588.010838:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00010000:00080000:7.0:1547251588.010842:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df2100 x1622384012444112/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251595 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251588.026713:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444128, portal 4 00000100:00000200:6.0:1547251588.026715:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444128, offset 0 00000400:00000200:6.0:1547251588.026718:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251588.026721:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251588.026723:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c041e0 00000400:00000200:6.0:1547251588.026726:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe1fd5 [1] + 0 00000400:00000200:6.0:1547251588.026728:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad000 00000100:00000200:6.0:1547251588.026730:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251588.026738:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad000 00000400:00000200:6.0:1547251588.026740:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251588.026744:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37934e00 x1622384012444128/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251595 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251588.026750:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459903b8 00000400:00000200:6.0:1547251588.026751:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251588.026753:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251588.026762:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444128 00010000:00000200:7.0:1547251588.026772:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39e850 x1622384012444128/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:649/0 lens 224/368 e 0 to 0 dl 1547251594 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:7.0:1547251588.026779:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444128, offset 192 00000400:00000200:7.0:1547251588.026781:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251588.026784:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251588.026786:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c041e0 00000400:00000200:7.0:1547251588.026788:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaec81 [1] + 192 00000100:00000200:7.0:1547251588.026792:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37934e00 x1622384012444128/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251595 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251588.026800:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251588.026802:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea7f8 00000400:00000200:7.0:1547251588.026803:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251588.026806:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251588.026810:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37934e00 x1622384012444128/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251595 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251588.026813:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251588.026816:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37934e00 x1622384012444128/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251595 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251588.026821:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990cc0 00010000:00080000:6.0:1547251588.026825:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37934e00 x1622384012444128/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251595 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251590.042734:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444144, portal 4 00000100:00000200:9.0:1547251590.042737:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444144, offset 0 00000400:00000200:9.0:1547251590.042741:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:9.0:1547251590.042744:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:9.0:1547251590.042747:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23015] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251590.042750:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23013] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23015] 00000400:00000200:9.0:1547251590.042752:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251590.042753:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251590.042755:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251590.042756:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9735, 9736] 00000400:00000200:9.0:1547251590.042757:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000100:00000200:0.0:1547251590.042759:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444160, portal 4 00000100:00000200:0.0:1547251590.042761:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444160, offset 0 00000400:00000200:9.0:1547251590.042762:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:0.0:1547251590.042764:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:9.0:1547251590.042765:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:0.0:1547251590.042767:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000800:00000200:9.0:1547251590.042768:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251590.042769:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000400:00000200:0.0:1547251590.042769:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23015] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:9.0:1547251590.042770:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:9.0:1547251590.042772:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:0.0:1547251590.042773:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23014] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23015] 00000400:00000200:0.0:1547251590.042775:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:0.0:1547251590.042777:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251590.042778:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251590.042779:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9736, 9736] 00000800:00000200:9.2:1547251590.042781:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:0.0:1547251590.042781:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:0.0:1547251590.042786:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251590.042789:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:0.0:1547251590.042792:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251590.042793:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:0.0:1547251590.042794:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:0.0:1547251590.042798:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:7.2:1547251590.042855:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251590.042866:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000100:00000200:6.0:1547251590.042866:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444176, portal 4 00000800:00000200:12.0:1547251590.042869:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000100:00000200:6.0:1547251590.042870:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444176, offset 0 00000100:00000200:12.0:1547251590.042872:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960eef00 x1622384012444144/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251597 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251590.042874:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:6.0:1547251590.042878:0:12675:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:6.0:1547251590.042881:0:12675:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23016] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251590.042883:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaaa0 00000400:00000200:12.0:1547251590.042884:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251590.042886:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:6.0:1547251590.042886:0:12675:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23014] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23016] 00000800:00000200:12.0:1547251590.042889:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:6.0:1547251590.042890:0:12675:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:12.0:1547251590.042891:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:12.0:1547251590.042892:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000400:00000200:6.0:1547251590.042892:0:12675:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:12.0:1547251590.042894:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c4800 x1622384012444160/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251597 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251590.042895:0:12675:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251590.042897:0:12675:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9736, 9737] 00000400:00000200:12.0:1547251590.042898:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00000400:00000200:6.0:1547251590.042898:0:12675:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251590.042899:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251590.042901:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:6.0:1547251590.042903:0:12675:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251590.042906:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:6.0:1547251590.042906:0:12675:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:12.0:1547251590.042907:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:12.0:1547251590.042908:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:6.0:1547251590.042910:0:12675:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:6.0:1547251590.042911:0:12675:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:6.0:1547251590.042912:0:12675:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:6.0:1547251590.042915:0:12675:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:9.2:1547251590.042928:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251590.042946:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251590.042954:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000100:00000200:12.0:1547251590.042956:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824448900 x1622384012444176/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251597 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251590.042960:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459903b8 00000800:00000200:14.0:1547251590.042961:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:12.0:1547251590.042961:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251590.042963:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251590.042965:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:9.2:1547251590.042983:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:7.2:1547251590.042988:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:14.0:1547251590.042995:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:14.0:1547251590.042998:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.93@o2ib10 00000800:00000200:12.0:1547251590.043001:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000400:00000200:14.0:1547251590.043003:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251590.043003:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251590.043007:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c041f0 00000400:00000200:12.0:1547251590.043008:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251590.043010:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7f9fd [1] + 192 00000400:00000200:12.0:1547251590.043012:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04200 00000100:00000200:14.0:1547251590.043014:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960eef00 x1622384012444144/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251597 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251590.043024:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251590.043026:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000400:00000200:12.0:1547251590.043026:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fa0d [1] + 192 00000800:00000200:14.0:1547251590.043028:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:14.0:1547251590.043029:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:14.0:1547251590.043030:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000100:00000200:9.0:1547251590.043030:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960eef00 x1622384012444144/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251597 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251590.043032:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251590.043035:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000100:00000200:9.0:1547251590.043035:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960eef00 x1622384012444144/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251597 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251590.043037:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04210 00000400:00000200:9.0:1547251590.043039:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea440 00000100:00000200:12.0:1547251590.043042:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c4800 x1622384012444160/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251597 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251590.043044:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba960eef00 x1622384012444144/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251597 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:12.0:1547251590.043051:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:14.0:1547251590.043052:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fa1d [1] + 192 00000800:00000200:12.0:1547251590.043052:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000100:00000200:14.0:1547251590.043054:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824448900 x1622384012444176/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251597 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251590.043054:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:12.0:1547251590.043055:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251590.043057:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:12.0:1547251590.043058:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:14.0:1547251590.043063:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251590.043064:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:14.0:1547251590.043065:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:0.0:1547251590.043065:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c4800 x1622384012444160/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251597 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251590.043066:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000100:00000200:0.0:1547251590.043071:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c4800 x1622384012444160/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251597 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251590.043075:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665198 00000100:00000200:6.0:1547251590.043078:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824448900 x1622384012444176/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251597 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251590.043078:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c4800 x1622384012444160/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251597 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251590.043084:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824448900 x1622384012444176/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251597 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251590.043089:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990cc0 00010000:00080000:6.0:1547251590.043092:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824448900 x1622384012444176/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251597 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251592.154678:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444192, portal 4 00000100:00000200:9.0:1547251592.154681:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444192, offset 0 00000400:00000200:9.0:1547251592.154684:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251592.154687:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251592.154689:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23016] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:15.0:1547251592.154690:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444208, portal 4 00000400:00000200:9.0:1547251592.154692:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23015] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23016] 00000100:00000200:15.0:1547251592.154693:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444208, offset 0 00000400:00000200:9.0:1547251592.154695:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:15.0:1547251592.154696:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251592.154696:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251592.154698:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251592.154699:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9741, 9742] 00000400:00000200:15.0:1547251592.154700:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251592.154701:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:15.0:1547251592.154702:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23016] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251592.154705:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000400:00000200:15.0:1547251592.154706:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23016] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23016] 00000400:00000200:15.0:1547251592.154708:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:9.0:1547251592.154708:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:15.0:1547251592.154709:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251592.154711:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251592.154711:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000400:00000200:15.0:1547251592.154712:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9742, 9742] 00000800:00000200:9.0:1547251592.154712:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:9.0:1547251592.154713:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000400:00000200:15.0:1547251592.154714:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000800:00000200:9.0:1547251592.154716:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000400:00000200:15.0:1547251592.154719:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251592.154722:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:15.0:1547251592.154725:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251592.154726:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:15.0:1547251592.154727:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:15.0:1547251592.154730:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000100:00000200:3.0:1547251592.154733:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444224, portal 4 00000100:00000200:3.0:1547251592.154736:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444224, offset 0 00000400:00000200:3.0:1547251592.154741:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:3.0:1547251592.154744:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:3.0:1547251592.154746:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23017] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251592.154749:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23016] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23017] 00000400:00000200:3.0:1547251592.154752:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:3.0:1547251592.154753:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251592.154755:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251592.154756:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9742, 9743] 00000400:00000200:3.0:1547251592.154758:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:3.0:1547251592.154762:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:7.2:1547251592.154765:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000800:00000200:3.0:1547251592.154765:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:3.0:1547251592.154768:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251592.154769:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (22)++ 00000800:00000200:3.0:1547251592.154770:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (23)++ 00000800:00000200:3.0:1547251592.154773:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (24)-- 00000800:00000200:5.2:1547251592.154784:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251592.154795:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:11.0:1547251592.154800:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000800:00000200:10.0:1547251592.154801:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251592.154803:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000100:00000200:11.0:1547251592.154804:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf6c00 x1622384012444192/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:10.0:1547251592.154806:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8caec00 x1622384012444208/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251592.154812:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea990 00000400:00000200:11.0:1547251592.154813:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251592.154813:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb12a8 00000400:00000200:10.0:1547251592.154814:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251592.154817:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251592.154817:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251592.154820:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251592.154821:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251592.154822:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251592.154823:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251592.154823:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:10.0:1547251592.154824:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000100:00000200:11.0:1547251592.154826:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b6c00 x1622384012444224/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251592.154832:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555bb0 00000400:00000200:11.0:1547251592.154833:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251592.154836:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251592.154838:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:7.2:1547251592.154899:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:5.2:1547251592.154905:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251592.154910:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:10.0:1547251592.154914:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251592.154918:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251592.154918:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251592.154920:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251592.154922:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251592.154925:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251592.154927:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04230 00000400:00000200:11.0:1547251592.154929:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04240 00000400:00000200:10.0:1547251592.154931:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6ae9 [1] + 192 00000400:00000200:11.0:1547251592.154936:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6af9 [1] + 192 00000100:00000200:10.0:1547251592.154949:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8caec00 x1622384012444208/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251592.154961:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251592.154962:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b6c00 x1622384012444224/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251592.154963:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251592.154965:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251592.154966:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251592.154968:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251592.154971:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251592.154973:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251592.154975:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000400:00000200:10.0:1547251592.154975:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000100:00000200:15.0:1547251592.154976:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8caec00 x1622384012444208/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251592.154977:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251592.154978:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:10.0:1547251592.154979:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04220 00000800:00000200:11.0:1547251592.154980:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:15.0:1547251592.154981:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8caec00 x1622384012444208/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251592.154982:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:11.0:1547251592.154983:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:15.0:1547251592.154985:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1330 00000400:00000200:10.0:1547251592.154988:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6ac9 [1] + 192 00010000:00080000:15.0:1547251592.154989:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8caec00 x1622384012444208/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251592.154993:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b6c00 x1622384012444224/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251592.154998:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b6c00 x1622384012444224/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251592.155007:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00000100:00000200:10.0:1547251592.155010:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf6c00 x1622384012444192/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:3.0:1547251592.155011:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b6c00 x1622384012444224/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251592.155018:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251592.155019:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251592.155020:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251592.155021:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:9.0:1547251592.155065:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf6c00 x1622384012444192/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251592.155071:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf6c00 x1622384012444192/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251592.155075:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea440 00010000:00080000:9.0:1547251592.155079:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf6c00 x1622384012444192/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:1.0:1547251592.618636:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444240, portal 4 00000100:00000200:1.0:1547251592.618639:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444240, offset 0 00000400:00000200:1.0:1547251592.618642:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:1.0:1547251592.618645:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:1.0:1547251592.618647:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23017] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251592.618650:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23017] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23017] 00000400:00000200:1.0:1547251592.618653:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:1.0:1547251592.618654:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251592.618656:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251592.618657:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9734, 9736] 00000400:00000200:1.0:1547251592.618659:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:1.0:1547251592.618663:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000100:00000200:8.0:1547251592.618665:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444256, portal 4 00000800:00000200:1.0:1547251592.618666:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000100:00000200:8.0:1547251592.618668:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444256, offset 0 00000800:00000200:1.0:1547251592.618669:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251592.618670:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000400:00000200:8.0:1547251592.618671:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:1.0:1547251592.618671:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000400:00000200:8.0:1547251592.618674:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:1.0:1547251592.618674:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000400:00000200:8.0:1547251592.618676:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23018] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251592.618679:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23017] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23018] 00000800:00000200:12.2:1547251592.618682:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000400:00000200:8.0:1547251592.618682:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:8.0:1547251592.618683:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251592.618684:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251592.618686:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9735, 9736] 00000400:00000200:8.0:1547251592.618688:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:8.0:1547251592.618692:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251592.618697:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:8.0:1547251592.618700:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251592.618701:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:8.0:1547251592.618702:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:8.0:1547251592.618705:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:0.2:1547251592.618720:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000100:00000200:15.0:1547251592.618728:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444272, portal 4 00000100:00000200:15.0:1547251592.618732:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444272, offset 0 00000400:00000200:15.0:1547251592.618736:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:15.0:1547251592.618739:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:5.0:1547251592.618740:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000400:00000200:15.0:1547251592.618743:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23018] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:5.0:1547251592.618744:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000800:00000200:4.0:1547251592.618744:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000400:00000200:15.0:1547251592.618746:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23018] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23018] 00000800:00000200:4.0:1547251592.618746:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000400:00000200:15.0:1547251592.618748:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000100:00000200:5.0:1547251592.618748:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b7200 x1622384012444240/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251592.618750:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:4.0:1547251592.618750:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824753900 x1622384012444256/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251592.618751:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251592.618752:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9735, 9737] 00000400:00000200:15.0:1547251592.618754:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251592.618756:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e26e8 00000400:00000200:4.0:1547251592.618756:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000400:00000200:5.0:1547251592.618757:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251592.618757:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:15.0:1547251592.618759:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251592.618761:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:5.0:1547251592.618761:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251592.618761:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:15.0:1547251592.618764:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:4.0:1547251592.618764:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251592.618765:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:15.0:1547251592.618766:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251592.618766:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:15.0:1547251592.618767:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251592.618767:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:15.0:1547251592.618770:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:12.2:1547251592.618783:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251592.618795:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251592.618803:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000100:00000200:4.0:1547251592.618807:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af0000 x1622384012444272/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251592.618810:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:4.0:1547251592.618813:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1a18 00000400:00000200:4.0:1547251592.618815:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251592.618818:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251592.618820:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:0.2:1547251592.618855:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251592.618865:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:5.0:1547251592.618872:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251592.618876:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251592.618877:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251592.618880:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251592.618881:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251592.618886:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04250 00000400:00000200:4.0:1547251592.618886:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251592.618890:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3885 [1] + 192 00000400:00000200:4.0:1547251592.618890:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04260 00000400:00000200:4.0:1547251592.618894:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3895 [1] + 192 00000100:00000200:5.0:1547251592.618898:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b7200 x1622384012444240/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251592.618909:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251592.618911:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000100:00000200:4.0:1547251592.618911:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824753900 x1622384012444256/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251592.618914:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251592.618915:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251592.618917:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251592.618920:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251592.618921:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251592.618923:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000100:00000200:1.0:1547251592.618923:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b7200 x1622384012444240/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251592.618924:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251592.618925:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251592.618926:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:5.0:1547251592.618927:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04270 00000800:00000200:4.0:1547251592.618927:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:4.0:1547251592.618929:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:1.0:1547251592.618929:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b7200 x1622384012444240/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251592.618938:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00000100:00000200:8.0:1547251592.618948:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824753900 x1622384012444256/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:1.0:1547251592.618950:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b7200 x1622384012444240/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251592.618954:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824753900 x1622384012444256/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251592.618958:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688a18 00010000:00080000:8.0:1547251592.618961:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824753900 x1622384012444256/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251592.618963:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe38a5 [1] + 192 00000100:00000200:5.0:1547251592.618967:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af0000 x1622384012444272/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251592.618976:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251592.618977:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251592.618978:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251592.618979:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:15.0:1547251592.618986:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af0000 x1622384012444272/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:15.0:1547251592.618991:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af0000 x1622384012444272/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251592.618996:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1330 00010000:00080000:15.0:1547251592.619000:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af0000 x1622384012444272/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251599 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251593.018676:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444288, portal 4 00000100:00000200:8.0:1547251593.018679:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444288, offset 0 00000400:00000200:8.0:1547251593.018681:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251593.018684:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251593.018687:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04280 00000400:00000200:8.0:1547251593.018689:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd43f9 [1] + 0 00000400:00000200:8.0:1547251593.018691:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b8245ea770 00000100:00000200:8.0:1547251593.018693:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251593.018702:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea770 00000400:00000200:8.0:1547251593.018703:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251593.018707:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824753300 x1622384012444288/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251600 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251593.018713:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000400:00000200:8.0:1547251593.018714:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251593.018716:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251593.018724:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444288 00000100:00000200:6.0:1547251593.018725:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444304, portal 4 00000100:00000200:6.0:1547251593.018727:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444304, offset 0 00000400:00000200:6.0:1547251593.018730:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251593.018733:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00010000:00000200:9.0:1547251593.018734:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3dc450 x1622384012444288/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:654/0 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:6.0:1547251593.018736:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04290 00000400:00000200:6.0:1547251593.018738:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2105 [1] + 0 00000100:00000200:9.0:1547251593.018740:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444288, offset 192 00000400:00000200:6.0:1547251593.018741:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad550 00000400:00000200:9.0:1547251593.018742:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:6.0:1547251593.018742:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251593.018746:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251593.018749:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04280 00000400:00000200:9.0:1547251593.018751:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaecc1 [1] + 192 00000400:00000200:6.0:1547251593.018751:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad550 00000400:00000200:6.0:1547251593.018752:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251593.018754:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824753300 x1622384012444288/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251600 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251593.018756:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37934b00 x1622384012444304/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251600 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251593.018762:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251593.018762:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990088 00000400:00000200:6.0:1547251593.018763:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251593.018764:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea440 00000400:00000200:9.0:1547251593.018765:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251593.018765:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:9.0:1547251593.018768:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251593.018772:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824753300 x1622384012444288/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251600 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251593.018774:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251593.018774:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444304 00000100:00000200:8.0:1547251593.018778:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824753300 x1622384012444288/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251600 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251593.018783:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688a18 00010000:00000200:7.0:1547251593.018784:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39e450 x1622384012444304/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:654/0 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Interpret:/0/0 rc 0/0 00010000:00080000:8.0:1547251593.018788:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824753300 x1622384012444288/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251600 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251593.018791:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444304, offset 192 00000400:00000200:7.0:1547251593.018793:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251593.018796:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251593.018799:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04290 00000400:00000200:7.0:1547251593.018801:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaecd1 [1] + 192 00000100:00000200:7.0:1547251593.018804:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37934b00 x1622384012444304/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251600 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251593.018812:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251593.018816:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea3b8 00000400:00000200:7.0:1547251593.018818:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251593.018819:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251593.018823:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37934b00 x1622384012444304/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251600 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251593.018827:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251593.018830:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37934b00 x1622384012444304/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251600 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251593.018835:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990cc0 00010000:00080000:6.0:1547251593.018839:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37934b00 x1622384012444304/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251600 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251593.034699:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444320, portal 4 00000100:00000200:6.0:1547251593.034703:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444320, offset 0 00000400:00000200:6.0:1547251593.034706:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251593.034709:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251593.034711:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c042a0 00000400:00000200:6.0:1547251593.034714:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2135 [1] + 0 00000400:00000200:6.0:1547251593.034716:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073adbb0 00000100:00000200:6.0:1547251593.034718:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251593.034724:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adbb0 00000400:00000200:6.0:1547251593.034725:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251593.034729:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37935100 x1622384012444320/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251600 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251593.034735:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990088 00000400:00000200:6.0:1547251593.034736:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251593.034738:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251593.034747:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444320 00010000:00000200:7.0:1547251593.034757:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39e050 x1622384012444320/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:654/0 lens 224/368 e 0 to 0 dl 1547251599 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:7.0:1547251593.034765:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444320, offset 192 00000400:00000200:7.0:1547251593.034768:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251593.034771:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251593.034773:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c042a0 00000400:00000200:7.0:1547251593.034775:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaece1 [1] + 192 00000100:00000200:7.0:1547251593.034778:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37935100 x1622384012444320/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251600 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251593.034786:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251593.034789:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea4c8 00000400:00000200:7.0:1547251593.034790:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251593.034792:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251593.034796:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37935100 x1622384012444320/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251600 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251593.034799:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251593.034802:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37935100 x1622384012444320/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251600 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251593.034807:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990cc0 00010000:00080000:6.0:1547251593.034811:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37935100 x1622384012444320/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251600 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251595.050701:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444336, portal 4 00000100:00000200:7.0:1547251595.050704:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444336, offset 0 00000400:00000200:7.0:1547251595.050707:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:7.0:1547251595.050710:0:12674:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:7.0:1547251595.050713:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23019] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:7.0:1547251595.050716:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23018] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23019] 00000400:00000200:7.0:1547251595.050718:0:12674:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:7.0:1547251595.050719:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:1.0:1547251595.050719:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444352, portal 4 00000400:00000200:7.0:1547251595.050721:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251595.050722:0:12674:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9737, 9737] 00000100:00000200:1.0:1547251595.050722:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444352, offset 0 00000400:00000200:7.0:1547251595.050724:0:12674:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:1.0:1547251595.050725:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:1.0:1547251595.050728:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:7.0:1547251595.050729:0:12674:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000100:00000200:9.0:1547251595.050730:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444368, portal 4 00000400:00000200:1.0:1547251595.050730:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23019] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:7.0:1547251595.050732:0:12674:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000100:00000200:9.0:1547251595.050733:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444368, offset 0 00000400:00000200:1.0:1547251595.050734:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23019] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23019] 00000400:00000200:9.0:1547251595.050736:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:1.0:1547251595.050736:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251595.050738:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:1.0:1547251595.050738:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251595.050739:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:7.0:1547251595.050740:0:12674:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000400:00000200:9.0:1547251595.050741:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23020] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251595.050741:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9737, 9738] 00000800:00000200:7.0:1547251595.050742:0:12674:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.0:1547251595.050743:0:12674:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:1.0:1547251595.050743:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:9.0:1547251595.050744:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23019] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23020] 00000400:00000200:9.0:1547251595.050746:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:7.0:1547251595.050746:0:12674:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:9.0:1547251595.050747:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251595.050747:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:9.0:1547251595.050749:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251595.050750:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9738, 9738] 00000800:00000200:1.0:1547251595.050750:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:9.2:1547251595.050753:0:12680:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:1.0:1547251595.050753:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251595.050755:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:1.0:1547251595.050756:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:1.0:1547251595.050758:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:9.0:1547251595.050766:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:9.0:1547251595.050771:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251595.050774:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:7.2:1547251595.050775:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:9.0:1547251595.050778:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251595.050779:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.0:1547251595.050780:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:9.0:1547251595.050783:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:7.2:1547251595.050789:0:1664:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251595.050805:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251595.050808:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:12.0:1547251595.050812:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df4200 x1622384012444336/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251602 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251595.050817:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea6e8 00000400:00000200:12.0:1547251595.050818:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251595.050820:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251595.050822:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251595.050824:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251595.050825:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:12.0:1547251595.050827:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c4500 x1622384012444352/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251602 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251595.050831:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000400:00000200:12.0:1547251595.050832:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2330 00000400:00000200:12.0:1547251595.050832:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:14.0:1547251595.050834:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000400:00000200:12.0:1547251595.050834:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251595.050836:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:14.0:1547251595.050837:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960ec200 x1622384012444368/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251602 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251595.050837:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:12.0:1547251595.050839:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:12.0:1547251595.050840:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000400:00000200:14.0:1547251595.050843:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea088 00000400:00000200:14.0:1547251595.050844:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251595.050846:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251595.050849:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:7.2:1547251595.050920:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251595.050925:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.2:1547251595.050931:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:12.0:1547251595.050933:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251595.050936:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000800:00000200:14.0:1547251595.050939:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:12.0:1547251595.050948:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:14.0:1547251595.050949:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251595.050952:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c042d0 00000400:00000200:14.0:1547251595.050953:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251595.050955:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fa6d [1] + 192 00000400:00000200:14.0:1547251595.050957:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c042b0 00000100:00000200:12.0:1547251595.050958:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960ec200 x1622384012444368/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251602 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251595.050969:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:14.0:1547251595.050970:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fa4d [1] + 192 00000800:00000200:12.0:1547251595.050971:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000800:00000200:12.0:1547251595.050972:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000100:00000200:14.0:1547251595.050974:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df4200 x1622384012444336/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251602 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251595.050974:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251595.050976:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251595.050978:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251595.050980:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251595.050983:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c042c0 00000400:00000200:14.0:1547251595.050985:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251595.050987:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000100:00000200:9.0:1547251595.050987:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960ec200 x1622384012444368/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251602 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251595.050988:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:14.0:1547251595.050989:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:14.0:1547251595.050991:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:14.0:1547251595.050992:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:9.0:1547251595.050992:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960ec200 x1622384012444368/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251602 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251595.050993:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:9.0:1547251595.050996:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea550 00000400:00000200:12.0:1547251595.051000:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fa5d [1] + 192 00010000:00080000:9.0:1547251595.051001:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba960ec200 x1622384012444368/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251602 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251595.051004:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df4200 x1622384012444336/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251602 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251595.051009:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df4200 x1622384012444336/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251602 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251595.051013:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea088 00000100:00000200:12.0:1547251595.051015:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c4500 x1622384012444352/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251602 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:7.0:1547251595.051017:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a765df4200 x1622384012444336/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251602 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:12.0:1547251595.051023:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251595.051024:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251595.051025:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251595.051026:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000100:00000200:1.0:1547251595.051033:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c4500 x1622384012444352/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251602 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:1.0:1547251595.051039:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c4500 x1622384012444352/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251602 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251595.051043:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00010000:00080000:1.0:1547251595.051046:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c4500 x1622384012444352/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251602 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251597.162681:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444384, portal 4 00000100:00000200:8.0:1547251597.162683:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444384, offset 0 00000400:00000200:8.0:1547251597.162687:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251597.162689:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251597.162692:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23021] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251597.162695:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23019] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23021] 00000100:00000200:14.0:1547251597.162696:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444400, portal 4 00000400:00000200:8.0:1547251597.162697:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000100:00000200:14.0:1547251597.162699:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444400, offset 0 00000400:00000200:8.0:1547251597.162699:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251597.162700:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251597.162701:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9743, 9743] 00000400:00000200:14.0:1547251597.162702:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251597.162703:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:14.0:1547251597.162705:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:14.0:1547251597.162708:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23021] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251597.162708:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000400:00000200:14.0:1547251597.162711:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23020] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23021] 00000800:00000200:8.0:1547251597.162711:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:14.0:1547251597.162714:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:8.0:1547251597.162714:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000400:00000200:14.0:1547251597.162715:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251597.162715:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:8.0:1547251597.162716:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:14.0:1547251597.162717:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251597.162718:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9743, 9744] 00000800:00000200:8.0:1547251597.162719:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:14.0:1547251597.162720:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:14.0:1547251597.162724:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251597.162727:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:14.0:1547251597.162730:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251597.162732:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:14.0:1547251597.162733:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000800:00000200:14.0:1547251597.162735:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:7.2:1547251597.162767:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000100:00000200:2.0:1547251597.162768:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444416, portal 4 00000100:00000200:2.0:1547251597.162770:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444416, offset 0 00000400:00000200:2.0:1547251597.162774:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:2.0:1547251597.162776:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:2.0:1547251597.162779:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23022] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251597.162782:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23020] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23022] 00000400:00000200:2.0:1547251597.162784:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251597.162786:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251597.162787:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251597.162789:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9744, 9744] 00000400:00000200:2.0:1547251597.162790:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251597.162791:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000400:00000200:2.0:1547251597.162795:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251597.162798:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:11.0:1547251597.162800:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:5.2:1547251597.162801:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00000800:00000200:2.0:1547251597.162801:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251597.162802:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:2.0:1547251597.162804:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (23)++ 00000100:00000200:11.0:1547251597.162805:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824750f00 x1622384012444384/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:2.0:1547251597.162807:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (24)-- 00000800:00000200:10.0:1547251597.162808:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (24)-- 00000800:00000200:10.0:1547251597.162810:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000400:00000200:11.0:1547251597.162812:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000800:00000200:10.0:1547251597.162812:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000400:00000200:11.0:1547251597.162814:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:10.0:1547251597.162816:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a5100 x1622384012444400/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251597.162817:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:7.2:1547251597.162819:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:11.0:1547251597.162821:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)-- 00000400:00000200:10.0:1547251597.162822:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000800:00000200:11.0:1547251597.162823:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251597.162824:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000400:00000200:10.0:1547251597.162824:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251597.162826:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000400:00000200:10.0:1547251597.162827:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000100:00000200:11.0:1547251597.162829:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f6900 x1622384012444416/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251597.162829:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:10.0:1547251597.162830:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:11.0:1547251597.162834:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06ab28 00000400:00000200:11.0:1547251597.162835:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251597.162839:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251597.162840:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:7.2:1547251597.162900:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:5.2:1547251597.162905:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:10.0:1547251597.162913:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251597.162917:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251597.162919:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251597.162922:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251597.162922:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251597.162926:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251597.162926:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c042f0 00000400:00000200:11.0:1547251597.162930:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c042e0 00000400:00000200:10.0:1547251597.162931:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6b79 [1] + 192 00000400:00000200:11.0:1547251597.162935:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6b59 [1] + 192 00000100:00000200:10.0:1547251597.162949:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a5100 x1622384012444400/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251597.162960:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251597.162961:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824750f00 x1622384012444384/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251597.162962:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251597.162964:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251597.162965:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251597.162967:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:10.0:1547251597.162968:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251597.162971:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251597.162975:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251597.162978:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04300 00000400:00000200:11.0:1547251597.162979:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251597.162980:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (23)++ 00000100:00000200:14.0:1547251597.162981:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a5100 x1622384012444400/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251597.162982:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (24)-- 00000800:00000200:11.0:1547251597.162983:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251597.162985:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000100:00000200:14.0:1547251597.162991:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a5100 x1622384012444400/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251597.162995:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00000400:00000200:10.0:1547251597.162998:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6b89 [1] + 192 00010000:00080000:14.0:1547251597.162999:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a5100 x1622384012444400/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251597.163003:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824750f00 x1622384012444384/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251597.163009:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824750f00 x1622384012444384/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251597.163013:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688a18 00000100:00000200:10.0:1547251597.163015:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f6900 x1622384012444416/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251597.163017:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824750f00 x1622384012444384/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251597.163024:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251597.163025:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251597.163026:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251597.163027:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:2.0:1547251597.163033:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f6900 x1622384012444416/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251597.163038:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f6900 x1622384012444416/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251597.163042:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00010000:00080000:2.0:1547251597.163046:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f6900 x1622384012444416/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251597.626685:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444432, portal 4 00000100:00000200:9.0:1547251597.626688:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444432, offset 0 00000400:00000200:9.0:1547251597.626691:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251597.626693:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251597.626696:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23022] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251597.626699:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23021] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23022] 00000400:00000200:9.0:1547251597.626701:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000100:00000200:0.0:1547251597.626701:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444448, portal 4 00000400:00000200:9.0:1547251597.626702:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251597.626704:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:0.0:1547251597.626704:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444448, offset 0 00000100:00000200:14.0:1547251597.626705:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444464, portal 4 00000400:00000200:9.0:1547251597.626705:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9736, 9737] 00000400:00000200:9.0:1547251597.626707:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:0.0:1547251597.626707:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000100:00000200:14.0:1547251597.626708:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444464, offset 0 00000400:00000200:0.0:1547251597.626710:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:14.0:1547251597.626711:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251597.626711:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:0.0:1547251597.626712:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23022] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:14.0:1547251597.626714:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:9.0:1547251597.626714:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:0.0:1547251597.626715:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23022] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23022] 00000400:00000200:14.0:1547251597.626716:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23022] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:9.0:1547251597.626717:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:0.0:1547251597.626718:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:9.0:1547251597.626719:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000400:00000200:0.0:1547251597.626719:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251597.626720:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23022] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23023] 00000800:00000200:9.0:1547251597.626720:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:0.0:1547251597.626721:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251597.626722:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:9.0:1547251597.626722:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:0.0:1547251597.626722:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9737, 9737] 00000400:00000200:14.0:1547251597.626723:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251597.626724:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:14.0:1547251597.626725:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251597.626726:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9737, 9738] 00000400:00000200:14.0:1547251597.626728:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:0.2:1547251597.626730:0:12670:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:0.0:1547251597.626741:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251597.626744:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:14.0:1547251597.626747:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251597.626747:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251597.626749:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:14.0:1547251597.626750:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:0.0:1547251597.626750:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:0.0:1547251597.626752:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:14.0:1547251597.626753:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251597.626755:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (21)++ 00000800:00000200:14.0:1547251597.626756:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (22)++ 00000800:00000200:14.0:1547251597.626759:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (23)-- 00000800:00000200:12.2:1547251597.626769:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251597.626783:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251597.626788:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:4.0:1547251597.626793:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf6f00 x1622384012444432/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251597.626795:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:5.0:1547251597.626797:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000400:00000200:4.0:1547251597.626800:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eac38 00000100:00000200:5.0:1547251597.626801:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328dd100 x1622384012444448/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251597.626802:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251597.626805:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:5.0:1547251597.626808:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665dd0 00000800:00000200:4.0:1547251597.626808:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:5.0:1547251597.626809:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251597.626810:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000400:00000200:5.0:1547251597.626812:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251597.626812:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:4.0:1547251597.626813:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000800:00000200:5.0:1547251597.626814:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251597.626816:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:4.0:1547251597.626816:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a6600 x1622384012444464/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251597.626822:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:4.0:1547251597.626823:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251597.626826:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251597.626828:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:12.2:1547251597.626910:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:0.2:1547251597.626915:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:5.0:1547251597.626924:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251597.626928:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000800:00000200:4.0:1547251597.626928:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:4.0:1547251597.626931:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251597.626933:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251597.626935:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251597.626937:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04320 00000400:00000200:4.0:1547251597.626939:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04310 00000400:00000200:4.0:1547251597.626951:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3925 [1] + 192 00000100:00000200:4.0:1547251597.626958:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf6f00 x1622384012444432/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251597.626971:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251597.626973:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3935 [1] + 192 00000800:00000200:4.0:1547251597.626973:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251597.626975:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:5.0:1547251597.626977:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328dd100 x1622384012444448/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251597.626977:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251597.626978:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251597.626981:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251597.626985:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251597.626988:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251597.626989:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04330 00000100:00000200:9.0:1547251597.626990:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf6f00 x1622384012444432/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251597.626990:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (23)++ 00000800:00000200:5.0:1547251597.626991:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (24)-- 00000800:00000200:5.0:1547251597.626993:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251597.626994:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:9.0:1547251597.626996:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf6f00 x1622384012444432/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251597.626996:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:9.0:1547251597.627000:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea550 00010000:00080000:9.0:1547251597.627003:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf6f00 x1622384012444432/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:4.0:1547251597.627003:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3945 [1] + 192 00000100:00000200:0.0:1547251597.627008:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328dd100 x1622384012444448/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251597.627013:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328dd100 x1622384012444448/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251597.627017:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665198 00000100:00000200:4.0:1547251597.627020:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a6600 x1622384012444464/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251597.627021:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328dd100 x1622384012444448/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:4.0:1547251597.627030:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251597.627031:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251597.627032:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251597.627033:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:14.0:1547251597.627085:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a6600 x1622384012444464/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:14.0:1547251597.627091:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a6600 x1622384012444464/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251597.627096:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00010000:00080000:14.0:1547251597.627100:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a6600 x1622384012444464/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251604 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251598.026675:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444480, portal 4 00000100:00000200:9.0:1547251598.026678:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444480, offset 0 00000400:00000200:9.0:1547251598.026681:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251598.026683:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251598.026686:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04340 00000400:00000200:9.0:1547251598.026689:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4479 [1] + 0 00000400:00000200:9.0:1547251598.026691:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b8245ea6e8 00000100:00000200:9.0:1547251598.026693:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251598.026701:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea6e8 00000400:00000200:9.0:1547251598.026702:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251598.026706:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf7200 x1622384012444480/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251605 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251598.026713:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eac38 00000400:00000200:9.0:1547251598.026713:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:7.0:1547251598.026714:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444496, portal 4 00000400:00000200:9.0:1547251598.026715:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251598.026717:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444496, offset 0 00000400:00000200:7.0:1547251598.026720:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251598.026723:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:8.0:1547251598.026724:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444480 00000400:00000200:7.0:1547251598.026726:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04350 00000400:00000200:7.0:1547251598.026728:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe21a5 [1] + 0 00000400:00000200:7.0:1547251598.026730:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad5d8 00000100:00000200:7.0:1547251598.026732:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00010000:00000200:8.0:1547251598.026734:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3dc050 x1622384012444480/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:659/0 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:8.0:1547251598.026740:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444480, offset 192 00000400:00000200:8.0:1547251598.026742:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251598.026742:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad5d8 00000400:00000200:7.0:1547251598.026743:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251598.026745:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df5d00 x1622384012444496/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251605 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251598.026746:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251598.026749:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04340 00000400:00000200:7.0:1547251598.026750:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea908 00000400:00000200:8.0:1547251598.026751:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaed21 [1] + 192 00000400:00000200:7.0:1547251598.026751:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251598.026752:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251598.026754:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf7200 x1622384012444480/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251605 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251598.026762:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251598.026764:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688a18 00000400:00000200:8.0:1547251598.026765:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:4.0:1547251598.026766:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444496 00000400:00000200:8.0:1547251598.026768:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251598.026772:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf7200 x1622384012444480/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251605 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251598.026774:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:9.0:1547251598.026778:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf7200 x1622384012444480/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251605 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00000200:4.0:1547251598.026778:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39dc50 x1622384012444496/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:659/0 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:9.0:1547251598.026783:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea550 00010000:00080000:9.0:1547251598.026787:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf7200 x1622384012444480/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251605 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251598.026787:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444496, offset 192 00000400:00000200:4.0:1547251598.026790:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251598.026794:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251598.026797:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04350 00000400:00000200:4.0:1547251598.026799:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaed31 [1] + 192 00000100:00000200:4.0:1547251598.026803:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df5d00 x1622384012444496/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251605 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251598.026813:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251598.026815:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b198 00000400:00000200:4.0:1547251598.026816:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251598.026817:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251598.026821:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df5d00 x1622384012444496/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251605 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251598.026823:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251598.026826:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df5d00 x1622384012444496/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251605 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251598.026830:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea088 00010000:00080000:7.0:1547251598.026833:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df5d00 x1622384012444496/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251605 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251598.042681:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444512, portal 4 00000100:00000200:7.0:1547251598.042684:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444512, offset 0 00000400:00000200:7.0:1547251598.042687:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251598.042690:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251598.042693:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04360 00000400:00000200:7.0:1547251598.042695:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe21d5 [1] + 0 00000400:00000200:7.0:1547251598.042697:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040b110 00000100:00000200:7.0:1547251598.042699:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:7.0:1547251598.042707:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b110 00000400:00000200:7.0:1547251598.042708:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251598.042710:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df5700 x1622384012444512/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251605 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251598.042715:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea908 00000400:00000200:7.0:1547251598.042716:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251598.042717:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251598.042738:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444512 00010000:00000200:4.0:1547251598.042750:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39d850 x1622384012444512/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:659/0 lens 224/368 e 0 to 0 dl 1547251604 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251598.042759:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444512, offset 192 00000400:00000200:4.0:1547251598.042762:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251598.042765:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251598.042768:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04360 00000400:00000200:4.0:1547251598.042771:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaed41 [1] + 192 00000100:00000200:4.0:1547251598.042774:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df5700 x1622384012444512/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251605 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251598.042784:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251598.042786:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b6e8 00000400:00000200:4.0:1547251598.042787:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251598.042788:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251598.042793:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df5700 x1622384012444512/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251605 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251598.042794:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251598.042797:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df5700 x1622384012444512/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251605 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251598.042801:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea088 00010000:00080000:7.0:1547251598.042804:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df5700 x1622384012444512/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251605 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251600.058676:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444528, portal 4 00000100:00000200:8.0:1547251600.058679:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444528, offset 0 00000400:00000200:8.0:1547251600.058682:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:8.0:1547251600.058685:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:8.0:1547251600.058687:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23024] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251600.058690:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23022] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23024] 00000400:00000200:8.0:1547251600.058692:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:8.0:1547251600.058694:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251600.058695:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251600.058697:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9738, 9739] 00000400:00000200:8.0:1547251600.058698:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:8.0:1547251600.058703:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251600.058706:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:8.0:1547251600.058709:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251600.058710:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:8.0:1547251600.058711:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:8.0:1547251600.058713:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:0.0:1547251600.058727:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444544, portal 4 00000100:00000200:0.0:1547251600.058730:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444544, offset 0 00000800:00000200:9.2:1547251600.058731:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:0.0:1547251600.058733:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:0.0:1547251600.058736:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:0.0:1547251600.058738:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23024] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251600.058741:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23023] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23024] 00000400:00000200:0.0:1547251600.058743:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:0.0:1547251600.058745:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251600.058746:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251600.058748:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9739, 9739] 00000400:00000200:0.0:1547251600.058750:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:0.0:1547251600.058754:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251600.058757:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:0.0:1547251600.058760:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251600.058761:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:0.0:1547251600.058762:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:0.0:1547251600.058765:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000100:00000200:4.0:1547251600.058770:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444560, portal 4 00000100:00000200:4.0:1547251600.058774:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444560, offset 0 00000400:00000200:4.0:1547251600.058778:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:7.2:1547251600.058780:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000400:00000200:4.0:1547251600.058781:0:12677:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:4.0:1547251600.058784:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23025] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:4.0:1547251600.058787:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23023] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23025] 00000800:00000200:14.0:1547251600.058788:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000400:00000200:4.0:1547251600.058789:0:12677:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:4.0:1547251600.058791:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:14.0:1547251600.058792:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000400:00000200:4.0:1547251600.058792:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251600.058793:0:12677:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9739, 9740] 00000400:00000200:4.0:1547251600.058795:0:12677:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000100:00000200:14.0:1547251600.058796:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824754500 x1622384012444528/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251607 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251600.058800:0:12677:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:14.0:1547251600.058801:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000400:00000200:14.0:1547251600.058802:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251600.058803:0:12677:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:14.0:1547251600.058805:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:4.0:1547251600.058806:0:12677:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251600.058807:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:4.0:1547251600.058807:0:12677:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:4.0:1547251600.058808:0:12677:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:14.0:1547251600.058809:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:14.0:1547251600.058810:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000100:00000200:14.0:1547251600.058812:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328dd400 x1622384012444544/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251607 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251600.058812:0:12677:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (23)-- 00000400:00000200:14.0:1547251600.058817:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665d48 00000400:00000200:14.0:1547251600.058817:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251600.058819:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251600.058821:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:14.0:1547251600.058822:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:14.0:1547251600.058823:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000100:00000200:14.0:1547251600.058825:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37934500 x1622384012444560/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251607 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251600.058829:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040be58 00000400:00000200:14.0:1547251600.058829:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251600.058831:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251600.058833:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251600.058834:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:14.0:1547251600.058837:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:7.2:1547251600.058897:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:9.2:1547251600.058902:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:9.2:1547251600.058909:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:12.0:1547251600.058910:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:14.0:1547251600.058914:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251600.058914:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000800:00000200:14.0:1547251600.058917:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251600.058917:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251600.058920:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251600.058922:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04390 00000400:00000200:14.0:1547251600.058924:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04370 00000400:00000200:12.0:1547251600.058925:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fabd [1] + 192 00000400:00000200:14.0:1547251600.058928:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fa9d [1] + 192 00000100:00000200:12.0:1547251600.058933:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37934500 x1622384012444560/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251607 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251600.058952:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:14.0:1547251600.058954:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824754500 x1622384012444528/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251607 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251600.058954:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251600.058956:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251600.058957:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251600.058959:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251600.058961:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251600.058964:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:12.0:1547251600.058964:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:14.0:1547251600.058966:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000100:00000200:4.0:1547251600.058966:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37934500 x1622384012444560/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251607 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251600.058967:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04380 00000800:00000200:14.0:1547251600.058968:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:14.0:1547251600.058969:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:14.0:1547251600.058971:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:14.0:1547251600.058972:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000100:00000200:4.0:1547251600.058972:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37934500 x1622384012444560/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251607 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251600.058973:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:4.0:1547251600.058976:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b440 00000400:00000200:12.0:1547251600.058978:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7faad [1] + 192 00010000:00080000:4.0:1547251600.058980:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ae37934500 x1622384012444560/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251607 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251600.058982:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824754500 x1622384012444528/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251607 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251600.058988:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824754500 x1622384012444528/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251607 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251600.058992:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000100:00000200:12.0:1547251600.058995:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328dd400 x1622384012444544/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251607 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251600.058997:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824754500 x1622384012444528/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251607 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:12.0:1547251600.059007:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251600.059009:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251600.059010:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251600.059011:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:0.0:1547251600.059018:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328dd400 x1622384012444544/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251607 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251600.059023:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328dd400 x1622384012444544/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251607 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251600.059028:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665198 00010000:00080000:0.0:1547251600.059031:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328dd400 x1622384012444544/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251607 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251602.170700:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444592, portal 4 00000100:00000200:9.0:1547251602.170703:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444592, offset 0 00000400:00000200:9.0:1547251602.170706:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251602.170709:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251602.170711:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23025] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:12.0:1547251602.170713:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444576, portal 4 00000400:00000200:9.0:1547251602.170714:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23024] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23025] 00000400:00000200:9.0:1547251602.170716:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000100:00000200:12.0:1547251602.170717:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444576, offset 0 00000400:00000200:9.0:1547251602.170717:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251602.170719:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251602.170720:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9744, 9745] 00000400:00000200:12.0:1547251602.170721:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251602.170722:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:12.0:1547251602.170724:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:12.0:1547251602.170726:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23025] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251602.170726:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251602.170729:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:12.0:1547251602.170730:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23025] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23025] 00000400:00000200:12.0:1547251602.170732:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:9.0:1547251602.170732:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000400:00000200:12.0:1547251602.170733:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251602.170734:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000400:00000200:12.0:1547251602.170735:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251602.170735:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000400:00000200:12.0:1547251602.170736:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9745, 9745] 00000400:00000200:12.0:1547251602.170738:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000800:00000200:9.0:1547251602.170738:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000400:00000200:12.0:1547251602.170742:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000100:00000200:3.0:1547251602.170743:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444608, portal 4 00000800:00000200:12.0:1547251602.170745:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000100:00000200:3.0:1547251602.170746:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444608, offset 0 00000800:00000200:12.0:1547251602.170748:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000400:00000200:3.0:1547251602.170749:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:12.0:1547251602.170750:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:12.0:1547251602.170751:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000400:00000200:3.0:1547251602.170752:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:12.0:1547251602.170754:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000400:00000200:3.0:1547251602.170754:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23026] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251602.170757:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23025] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23026] 00000400:00000200:3.0:1547251602.170759:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:3.0:1547251602.170761:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251602.170762:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251602.170764:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9745, 9746] 00000400:00000200:3.0:1547251602.170766:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:3.0:1547251602.170770:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251602.170773:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:3.0:1547251602.170776:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251602.170777:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (21)++ 00000800:00000200:3.0:1547251602.170778:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (22)++ 00000800:00000200:3.0:1547251602.170781:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (23)-- 00000800:00000200:7.2:1547251602.170788:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (22)++ 00000800:00000200:5.2:1547251602.170808:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251602.170819:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:11.0:1547251602.170824:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000800:00000200:10.0:1547251602.170825:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000100:00000200:11.0:1547251602.170828:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf7500 x1622384012444592/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251602.170828:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000100:00000200:10.0:1547251602.170831:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8ca8600 x1622384012444576/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251602.170836:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eac38 00000400:00000200:11.0:1547251602.170837:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251602.170837:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038dd0 00000400:00000200:10.0:1547251602.170839:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251602.170841:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251602.170842:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251602.170844:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251602.170845:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251602.170846:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251602.170847:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251602.170847:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:10.0:1547251602.170848:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000100:00000200:11.0:1547251602.170850:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c4200 x1622384012444608/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251602.170856:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245555d8 00000400:00000200:11.0:1547251602.170857:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251602.170860:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251602.170862:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:7.2:1547251602.170911:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:5.2:1547251602.170917:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251602.170922:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:10.0:1547251602.170925:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251602.170929:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251602.170930:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251602.170932:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251602.170934:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251602.170937:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251602.170939:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c043a0 00000400:00000200:11.0:1547251602.170949:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c043b0 00000400:00000200:10.0:1547251602.170952:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6c09 [1] + 192 00000400:00000200:11.0:1547251602.170956:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6be9 [1] + 192 00000100:00000200:10.0:1547251602.170960:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8ca8600 x1622384012444576/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251602.170972:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251602.170973:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf7500 x1622384012444592/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251602.170974:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251602.170976:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251602.170977:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251602.170979:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251602.170981:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251602.170984:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:12.0:1547251602.170986:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8ca8600 x1622384012444576/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251602.170986:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000400:00000200:10.0:1547251602.170986:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251602.170988:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251602.170989:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:10.0:1547251602.170989:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c043c0 00000100:00000200:12.0:1547251602.170992:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8ca8600 x1622384012444576/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251602.170992:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:11.0:1547251602.170994:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:11.0:1547251602.170995:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:12.0:1547251602.170996:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038cc0 00000400:00000200:10.0:1547251602.170999:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6c19 [1] + 192 00010000:00080000:12.0:1547251602.171000:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8ca8600 x1622384012444576/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251602.171003:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf7500 x1622384012444592/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251602.171009:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf7500 x1622384012444592/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251602.171013:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea550 00000100:00000200:10.0:1547251602.171016:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c4200 x1622384012444608/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251602.171016:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf7500 x1622384012444592/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251602.171025:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251602.171026:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251602.171027:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251602.171028:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:3.0:1547251602.171034:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c4200 x1622384012444608/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251602.171039:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c4200 x1622384012444608/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251602.171044:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251602.171047:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c4200 x1622384012444608/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251602.634683:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444624, portal 4 00000100:00000200:8.0:1547251602.634686:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444624, offset 0 00000400:00000200:8.0:1547251602.634689:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251602.634692:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251602.634694:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23026] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251602.634697:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23026] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23026] 00000400:00000200:8.0:1547251602.634700:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251602.634701:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:15.0:1547251602.634703:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444656, portal 4 00000400:00000200:8.0:1547251602.634703:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251602.634704:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9737, 9739] 00000100:00000200:15.0:1547251602.634706:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444656, offset 0 00000400:00000200:8.0:1547251602.634706:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:15.0:1547251602.634709:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251602.634711:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:15.0:1547251602.634712:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:8.0:1547251602.634713:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:15.0:1547251602.634715:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23027] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:8.0:1547251602.634716:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:15.0:1547251602.634718:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23026] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23027] 00000800:00000200:8.0:1547251602.634718:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:8.0:1547251602.634719:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000400:00000200:15.0:1547251602.634720:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:8.0:1547251602.634721:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000400:00000200:15.0:1547251602.634722:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251602.634723:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251602.634724:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9738, 9739] 00000400:00000200:15.0:1547251602.634727:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:12.2:1547251602.634731:0:12684:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000400:00000200:15.0:1547251602.634732:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251602.634735:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:15.0:1547251602.634738:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251602.634739:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000100:00000200:3.0:1547251602.634739:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444640, portal 4 00000800:00000200:15.0:1547251602.634741:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:15.0:1547251602.634743:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000100:00000200:3.0:1547251602.634743:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444640, offset 0 00000400:00000200:3.0:1547251602.634748:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:3.0:1547251602.634751:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:3.0:1547251602.634753:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23027] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251602.634756:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23027] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23027] 00000800:00000200:0.2:1547251602.634758:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000400:00000200:3.0:1547251602.634759:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251602.634760:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251602.634762:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251602.634763:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9738, 9740] 00000400:00000200:3.0:1547251602.634765:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:3.0:1547251602.634769:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251602.634772:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:3.0:1547251602.634775:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251602.634777:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:3.0:1547251602.634778:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (23)++ 00000800:00000200:3.0:1547251602.634780:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (24)-- 00000800:00000200:5.0:1547251602.634790:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (23)++ 00000800:00000200:5.0:1547251602.634794:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (24)-- 00000800:00000200:4.0:1547251602.634794:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251602.634797:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000100:00000200:5.0:1547251602.634799:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824756f00 x1622384012444624/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251602.634800:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8caa400 x1622384012444656/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251602.634806:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00000400:00000200:4.0:1547251602.634807:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb17f8 00000400:00000200:5.0:1547251602.634808:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251602.634808:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251602.634812:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251602.634812:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251602.634814:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251602.634815:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251602.634817:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251602.634817:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251602.634818:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251602.634819:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000100:00000200:4.0:1547251602.634821:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b7500 x1622384012444640/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251602.634827:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555660 00000400:00000200:4.0:1547251602.634828:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251602.634831:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251602.634833:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:12.2:1547251602.634862:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:5.0:1547251602.634876:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251602.634885:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.1.11.92@o2ib10 00000800:00000200:5.0:1547251602.634887:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251602.634889:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:0.2:1547251602.634890:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:5.0:1547251602.634891:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251602.634892:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251602.634893:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251602.634895:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000800:00000200:5.0:1547251602.634896:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251602.634899:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251602.634900:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:0.2:1547251602.634900:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000400:00000200:5.0:1547251602.634905:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c043f0 00000400:00000200:4.0:1547251602.634905:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c043d0 00000400:00000200:4.0:1547251602.634909:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe39c5 [1] + 192 00000400:00000200:5.0:1547251602.634913:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe39d5 [1] + 192 00000100:00000200:4.0:1547251602.634917:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824756f00 x1622384012444624/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251602.634928:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251602.634930:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8caa400 x1622384012444656/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251602.634936:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251602.634938:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:4.0:1547251602.634939:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000100:00000200:8.0:1547251602.634949:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824756f00 x1622384012444624/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251602.634950:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:4.0:1547251602.634952:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000100:00000200:8.0:1547251602.634954:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824756f00 x1622384012444624/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251602.634954:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:4.0:1547251602.634957:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:8.0:1547251602.634958:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000400:00000200:5.0:1547251602.634958:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251602.634960:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000100:00000200:15.0:1547251602.634962:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8caa400 x1622384012444656/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251602.634962:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824756f00 x1622384012444624/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:5.0:1547251602.634962:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:4.0:1547251602.634962:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:5.0:1547251602.634963:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:5.0:1547251602.634965:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:4.0:1547251602.634965:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c043e0 00000100:00000200:15.0:1547251602.634968:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8caa400 x1622384012444656/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251602.634972:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1330 00000400:00000200:4.0:1547251602.634975:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe39e5 [1] + 192 00010000:00080000:15.0:1547251602.634976:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8caa400 x1622384012444656/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251602.634980:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b7500 x1622384012444640/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251602.634990:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251602.634991:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:4.0:1547251602.634993:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251602.634993:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:3.0:1547251602.635000:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b7500 x1622384012444640/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251602.635005:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b7500 x1622384012444640/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251602.635009:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251602.635013:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b7500 x1622384012444640/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251609 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251603.034677:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444672, portal 4 00000100:00000200:8.0:1547251603.034681:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444672, offset 0 00000400:00000200:8.0:1547251603.034683:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251603.034686:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251603.034689:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04400 00000400:00000200:8.0:1547251603.034692:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4569 [1] + 0 00000400:00000200:8.0:1547251603.034695:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ae6a9143b8 00000100:00000200:8.0:1547251603.034696:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251603.034705:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a9143b8 00000400:00000200:8.0:1547251603.034706:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251603.034710:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824755400 x1622384012444672/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251610 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251603.034717:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6883b8 00000400:00000200:8.0:1547251603.034717:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251603.034719:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251603.034727:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444672 00010000:00000200:9.0:1547251603.034737:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3dbc50 x1622384012444672/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:664/0 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:9.0:1547251603.034744:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444672, offset 192 00000100:00000200:4.0:1547251603.034744:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444688, portal 4 00000400:00000200:9.0:1547251603.034746:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251603.034748:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:4.0:1547251603.034748:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444688, offset 0 00000400:00000200:9.0:1547251603.034751:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04400 00000400:00000200:4.0:1547251603.034751:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251603.034753:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaed81 [1] + 192 00000400:00000200:4.0:1547251603.034755:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:9.0:1547251603.034756:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824755400 x1622384012444672/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251610 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251603.034757:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04410 00000400:00000200:4.0:1547251603.034760:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2245 [1] + 0 00000400:00000200:4.0:1547251603.034762:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990440 00000400:00000200:9.0:1547251603.034764:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251603.034764:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251603.034766:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea550 00000400:00000200:9.0:1547251603.034767:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251603.034770:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251603.034773:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990440 00000100:00000200:8.0:1547251603.034774:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824755400 x1622384012444672/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251610 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251603.034774:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251603.034776:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:4.0:1547251603.034777:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37936c00 x1622384012444688/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251610 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251603.034780:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824755400 x1622384012444672/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251610 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251603.034782:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bdd0 00000400:00000200:4.0:1547251603.034782:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251603.034784:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:8.0:1547251603.034786:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00010000:00080000:8.0:1547251603.034790:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824755400 x1622384012444672/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251610 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251603.034803:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444688 00010000:00000200:6.0:1547251603.034812:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39d450 x1622384012444688/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:664/0 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251603.034819:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444688, offset 192 00000400:00000200:6.0:1547251603.034822:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251603.034825:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251603.034829:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04410 00000400:00000200:6.0:1547251603.034831:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaed91 [1] + 192 00000100:00000200:6.0:1547251603.034834:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37936c00 x1622384012444688/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251610 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251603.034844:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251603.034846:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990cc0 00000400:00000200:6.0:1547251603.034846:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251603.034848:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251603.034852:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37936c00 x1622384012444688/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251610 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251603.034853:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:4.0:1547251603.034861:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37936c00 x1622384012444688/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251610 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251603.034865:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b440 00010000:00080000:4.0:1547251603.034868:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37936c00 x1622384012444688/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251610 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251603.050708:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444704, portal 4 00000100:00000200:5.0:1547251603.050711:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444704, offset 0 00000400:00000200:5.0:1547251603.050715:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251603.050718:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251603.050721:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04420 00000400:00000200:5.0:1547251603.050723:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2275 [1] + 0 00000400:00000200:5.0:1547251603.050726:0:12675:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990b28 00000100:00000200:5.0:1547251603.050728:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:5.0:1547251603.050737:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990b28 00000400:00000200:5.0:1547251603.050738:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251603.050741:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37931500 x1622384012444704/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251610 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251603.050746:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adcc0 00000400:00000200:5.0:1547251603.050746:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251603.050747:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251603.050767:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444704 00010000:00000200:6.0:1547251603.050776:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39d050 x1622384012444704/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:664/0 lens 224/368 e 0 to 0 dl 1547251609 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251603.050783:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444704, offset 192 00000400:00000200:6.0:1547251603.050785:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251603.050788:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251603.050792:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04420 00000400:00000200:6.0:1547251603.050794:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaeda1 [1] + 192 00000100:00000200:6.0:1547251603.050797:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37931500 x1622384012444704/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251610 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251603.050806:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251603.050809:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990088 00000400:00000200:6.0:1547251603.050809:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251603.050810:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251603.050813:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37931500 x1622384012444704/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251610 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251603.050816:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251603.050818:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37931500 x1622384012444704/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251610 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251603.050822:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad908 00010000:00080000:5.0:1547251603.050825:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37931500 x1622384012444704/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251610 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:0.2:1547251604.829359:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251604.829376:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:4.0:1547251604.829419:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251604.829422:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251604.829423:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000800:00000200:5.0:1547251604.829426:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251604.829429:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251604.829430:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251604.829434:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b102a0 00000400:00000200:4.0:1547251604.829434:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b102b0 00000400:00000200:4.0:1547251604.829438:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 59808 00000400:00000200:5.0:1547251604.829441:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 60032 00000100:00000200:4.0:1547251604.829445:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:5.0:1547251604.829456:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:4.0:1547251604.829456:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251604.829458:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251604.829461:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251604.829462:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251604.829463:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:5.0:1547251604.829465:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251604.829466:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000800:00000200:5.0:1547251604.829467:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251604.829468:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251604.829470:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:4.0:1547251604.829470:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:5.0:1547251604.829472:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000400:00000200:4.0:1547251604.829473:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 26 MB=0x5c38cb6b10280 00000800:00000200:5.0:1547251604.829474:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251604.829477:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0xd4d29 [1] + 0 00000400:00000200:5.0:1547251604.829478:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251604.829479:0:12661:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d110 00000100:00000200:4.0:1547251604.829481:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:5.0:1547251604.829482:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b10290 00000400:00000200:5.0:1547251604.829485:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 60256 00000100:00000200:5.0:1547251604.829487:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:4.0:1547251604.829488:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d110 00000400:00000200:4.0:1547251604.829489:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251604.829491:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251604.829493:0:12661:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (23)++ 00000400:00000200:5.0:1547251604.829495:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251604.829497:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251604.829497:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:4.0:1547251604.829498:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251604.829499:0:12662:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00000800:00000200:4.0:1547251604.829500:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251604.829501:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (24)-- 00000800:00000200:4.0:1547251604.829501:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (24)-- 00000800:00000200:5.0:1547251604.829502:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:0.2:1547251604.829504:0:12929:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:12.2:1547251604.829513:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000100:00000200:6.0:1547251604.829518:0:12979:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444912 00000100:00000200:0.0:1547251604.829518:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444864 00000100:00000200:7.0:1547251604.829519:0:13355:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444896 00000100:00000200:4.0:1547251604.829523:0:12981:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444880 00010000:00000200:0.0:1547251604.829524:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2d9050 x1622384011444864/t0(0) o400->04d65b15-cf9c-7ded-611d-ffe7fe3f3abf@10.0.11.92@o2ib10:665/0 lens 224/224 e 0 to 0 dl 1547251610 ref 1 fl Interpret:/0/0 rc 0/0 00010000:00000200:7.0:1547251604.829526:0:13355:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae5c433000 x1622384011444896/t0(0) o400->cache1-MDT0000-lwp-OST0006_UUID@10.0.11.92@o2ib10:665/0 lens 224/224 e 0 to 0 dl 1547251610 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:6.0:1547251604.829526:0:12979:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444d700 x1622384011444912/t0(0) o400->cache1-MDT0000-lwp-OST0008_UUID@10.0.11.92@o2ib10:665/0 lens 224/224 e 0 to 0 dl 1547251610 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:0.0:1547251604.829530:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384011444864, offset 192 00010000:00000200:4.0:1547251604.829531:0:12981:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae5c437b00 x1622384011444880/t0(0) o400->cache1-MDT0000-lwp-OST0007_UUID@10.0.11.92@o2ib10:665/0 lens 224/224 e 0 to 0 dl 1547251610 ref 1 fl Interpret:H/0/0 rc 0/0 00000800:00000200:5.0:1547251604.829532:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000400:00000200:0.0:1547251604.829533:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000800:00000200:5.0:1547251604.829534:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:7.0:1547251604.829535:0:13355:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444896, offset 192 00000100:00000200:6.0:1547251604.829535:0:12979:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444912, offset 192 00000800:00000200:5.0:1547251604.829535:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251604.829537:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000400:00000200:0.0:1547251604.829538:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00000400:00000200:7.0:1547251604.829539:0:13355:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:5.0:1547251604.829539:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251604.829540:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:0.0:1547251604.829540:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:5.0:1547251604.829541:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000400:00000200:0.0:1547251604.829541:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:5.0:1547251604.829543:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000400:00000200:0.0:1547251604.829543:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9739, 9740] 00000400:00000200:7.0:1547251604.829544:0:13355:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:6.0:1547251604.829544:0:12979:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000100:00000200:4.0:1547251604.829545:0:12981:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444880, offset 192 00000400:00000200:0.0:1547251604.829545:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:7.0:1547251604.829547:0:13355:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251604.829549:0:13355:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251604.829549:0:12981:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:7.0:1547251604.829551:0:13355:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9740, 9740] 00000400:00000200:7.0:1547251604.829555:0:13355:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251604.829562:0:13355:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:7.0:1547251604.829566:0:13355:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:6.0:1547251604.829567:0:12979:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00000400:00000200:6.0:1547251604.829569:0:12979:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251604.829572:0:12979:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251604.829573:0:12979:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9740, 9741] 00000400:00000200:6.0:1547251604.829576:0:12979:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:6.0:1547251604.829582:0:12979:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.1.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:6.0:1547251604.829603:0:12979:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:0.0:1547251604.829609:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.1.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:6.0:1547251604.829614:0:12979:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251604.829614:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:6.0:1547251604.829616:0:12979:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:6.0:1547251604.829617:0:12979:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:0.0:1547251604.829618:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251604.829620:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:6.0:1547251604.829621:0:12979:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:0.0:1547251604.829621:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (23)++ 00000400:00000200:4.0:1547251604.829625:0:12981:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:4.0:1547251604.829627:0:12981:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:0.2:1547251604.829627:0:12929:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (23)++ 00000400:00000200:4.0:1547251604.829628:0:12981:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251604.829630:0:12981:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[6, 7], s:[9741, 9741] 00000400:00000200:4.0:1547251604.829631:0:12981:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:0.0:1547251604.829634:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:4.0:1547251604.829640:0:12981:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000100:00000200:0.0:1547251604.829640:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:4.0:1547251604.829644:0:12981:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:7.0:1547251604.829645:0:13355:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:5.0:1547251604.829645:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)++ 00000800:00000200:7.0:1547251604.829647:0:13355:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:5.0:1547251604.829647:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:7.0:1547251604.829648:0:13355:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251604.829648:0:12981:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:5.0:1547251604.829649:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459905d8 00000400:00000200:5.0:1547251604.829650:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251604.829650:0:12981:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (22)++ 00000800:00000200:7.0:1547251604.829651:0:13355:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251604.829651:0:12981:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00000400:00000200:5.0:1547251604.829653:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251604.829654:0:12981:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251604.829656:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251604.829657:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251604.829659:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:5.0:1547251604.829661:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665198 00000800:00000200:12.2:1547251604.829662:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (22)++ 00000400:00000200:5.0:1547251604.829662:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251604.829664:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:5.0:1547251604.829665:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251604.829665:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:5.0:1547251604.829667:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:4.0:1547251604.829667:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000800:00000200:5.0:1547251604.829668:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000400:00000200:4.0:1547251604.829668:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea088 00000800:00000200:5.0:1547251604.829669:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000400:00000200:4.0:1547251604.829669:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251604.829670:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b440 00000400:00000200:5.0:1547251604.829671:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251604.829672:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251604.829674:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000400:00000200:5.0:1547251604.829675:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251604.829676:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251604.829677:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:7.2:1547251604.863694:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251604.863708:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:14.0:1547251604.863749:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:14.0:1547251604.863753:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251604.863757:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251604.863761:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb35102a0 00000400:00000200:14.0:1547251604.863764:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 29024 00000800:00000200:12.0:1547251604.863764:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000100:00000200:14.0:1547251604.863767:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:12.0:1547251604.863767:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251604.863772:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251604.863776:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510290 00000400:00000200:14.0:1547251604.863777:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251604.863779:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000400:00000200:12.0:1547251604.863779:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 29248 00000800:00000200:14.0:1547251604.863781:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000100:00000200:12.0:1547251604.863782:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:14.0:1547251604.863783:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:14.0:1547251604.863784:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:14.0:1547251604.863788:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251604.863791:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:14.0:1547251604.863793:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251604.863793:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251604.863795:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251604.863796:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251604.863797:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251604.863799:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251604.863802:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251604.863804:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb35102b0 00000400:00000200:12.0:1547251604.863806:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 29472 00000100:00000200:12.0:1547251604.863809:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:14.0:1547251604.863810:0:13374:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821776 00000400:00000200:12.0:1547251604.863813:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251604.863815:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00010000:00000200:14.0:1547251604.863816:0:13374:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97bae1af0f00 x1622383954821776/t0(0) o400->cache1-MDT0000-lwp-OST000a_UUID@10.0.11.93@o2ib10:665/0 lens 224/224 e 0 to 0 dl 1547251610 ref 1 fl Interpret:H/0/0 rc 0/0 00000800:00000200:12.0:1547251604.863816:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251604.863818:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251604.863820:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000100:00000200:15.0:1547251604.863821:0:12987:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821792 00000800:00000200:12.0:1547251604.863822:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000100:00000200:14.0:1547251604.863825:0:13374:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821776, offset 192 00010000:00000200:15.0:1547251604.863826:0:12987:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b0a8cad100 x1622383954821792/t0(0) o400->cache1-MDT0000-lwp-OST0009_UUID@10.0.11.93@o2ib10:665/0 lens 224/224 e 0 to 0 dl 1547251610 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:14.0:1547251604.863829:0:13374:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000100:00000200:12.0:1547251604.863829:0:12985:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821808 00000100:00000200:15.0:1547251604.863833:0:12987:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821792, offset 192 00000400:00000200:14.0:1547251604.863833:0:13374:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00000400:00000200:15.0:1547251604.863836:0:12987:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:14.0:1547251604.863836:0:13374:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00010000:00000200:12.0:1547251604.863836:0:12985:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97bae1af6900 x1622383954821808/t0(0) o400->cache1-MDT0000-lwp-OST000b_UUID@10.0.11.93@o2ib10:665/0 lens 224/224 e 0 to 0 dl 1547251610 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:14.0:1547251604.863838:0:13374:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251604.863840:0:13374:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9740, 9740] 00000400:00000200:14.0:1547251604.863842:0:13374:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000100:00000200:12.0:1547251604.863845:0:12985:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821808, offset 192 00000400:00000200:12.0:1547251604.863848:0:12985:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000400:00000200:14.0:1547251604.863849:0:13374:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.1.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251604.863852:0:13374:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:15.0:1547251604.863854:0:12987:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:15.0:1547251604.863856:0:12987:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251604.863858:0:12987:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251604.863860:0:12987:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9740, 9741] 00000400:00000200:15.0:1547251604.863862:0:12987:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251604.863869:0:12987:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251604.863872:0:12987:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:12.0:1547251604.863873:0:12985:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00000400:00000200:12.0:1547251604.863875:0:12985:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251604.863877:0:12985:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251604.863878:0:12985:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9741, 9741] 00000400:00000200:12.0:1547251604.863880:0:12985:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251604.863885:0:12985:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.1.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251604.863888:0:12985:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:15.0:1547251604.863889:0:12987:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251604.863889:0:13374:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251604.863891:0:12987:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:14.0:1547251604.863891:0:13374:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:12.0:1547251604.863891:0:12985:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251604.863892:0:12987:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251604.863893:0:12985:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:14.0:1547251604.863894:0:13374:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251604.863896:0:12987:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:14.0:1547251604.863896:0:13374:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:12.0:1547251604.863896:0:12985:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:12.0:1547251604.863899:0:12985:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (23)-- 00000400:00000200:14.0:1547251604.863906:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 26 MB=0x5c38cb3510280 00000800:00000200:7.2:1547251604.863909:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:14.0:1547251604.863910:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0xd4da9 [1] + 0 00000400:00000200:14.0:1547251604.863913:0:12666:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d198 00000800:00000200:9.2:1547251604.863914:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (22)++ 00000100:00000200:14.0:1547251604.863915:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000800:00000200:12.0:1547251604.863918:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251604.863919:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000400:00000200:12.0:1547251604.863921:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1330 00000400:00000200:12.0:1547251604.863922:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251604.863923:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d198 00000400:00000200:14.0:1547251604.863924:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:12.0:1547251604.863924:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251604.863926:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (21)++ 00000800:00000200:14.0:1547251604.863927:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251604.863927:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251604.863928:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:14.0:1547251604.863929:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:12.0:1547251604.863929:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (24)-- 00000400:00000200:12.0:1547251604.863930:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00000800:00000200:14.0:1547251604.863931:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000400:00000200:12.0:1547251604.863931:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:14.0:1547251604.863932:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)++ 00000400:00000200:12.0:1547251604.863933:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251604.863934:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (24)-- 00000800:00000200:12.0:1547251604.863934:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (24)-- 00000400:00000200:14.0:1547251604.863935:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038cc0 00000800:00000200:12.0:1547251604.863935:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:14.0:1547251604.863936:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251604.863938:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251604.863948:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:0.0:1547251604.863986:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821760 00010000:00000200:0.0:1547251604.863992:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2db050 x1622383954821760/t0(0) o400->b99b3dbb-ff3e-ba67-561d-3f0ab43da035@10.0.11.93@o2ib10:665/0 lens 224/224 e 0 to 0 dl 1547251610 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:0.0:1547251604.863998:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954821760, offset 192 00000400:00000200:0.0:1547251604.864001:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:0.0:1547251604.864005:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:0.0:1547251604.864007:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251604.864009:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251604.864010:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9741, 9742] 00000400:00000200:0.0:1547251604.864012:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:0.0:1547251604.864017:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251604.864020:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:0.0:1547251604.864023:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251604.864024:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:0.0:1547251604.864025:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:0.0:1547251604.864028:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:0.0:1547251604.864032:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:7.2:1547251604.864039:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251604.864051:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251604.864058:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000400:00000200:12.0:1547251604.864060:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665880 00000400:00000200:12.0:1547251604.864061:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251604.864063:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251604.864064:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251604.864065:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:5.2:1547251604.895385:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251604.895400:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:11.0:1547251604.895442:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251604.895446:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251604.895447:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251604.895449:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251604.895452:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251604.895454:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251604.895458:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb35102e0 00000400:00000200:10.0:1547251604.895458:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb35102d0 00000400:00000200:11.0:1547251604.895462:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 12768 00000400:00000200:10.0:1547251604.895467:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 12992 00000100:00000200:11.0:1547251604.895470:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:11.0:1547251604.895479:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251604.895480:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:11.0:1547251604.895482:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251604.895484:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251604.895485:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251604.895487:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251604.895488:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251604.895489:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251604.895490:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251604.895491:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251604.895492:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251604.895493:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000400:00000200:11.0:1547251604.895494:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251604.895496:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251604.895499:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251604.895502:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb35102c0 00000400:00000200:10.0:1547251604.895506:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 13216 00000100:00000200:10.0:1547251604.895508:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:10.0:1547251604.895517:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251604.895518:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251604.895519:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251604.895520:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000100:00000200:11.0:1547251604.895522:0:12983:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821856 00000800:00000200:10.0:1547251604.895522:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251604.895524:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00010000:00000200:11.0:1547251604.895528:0:12983:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ba960e8c00 x1622383954821856/t0(0) o400->cache1-MDT0000-lwp-OST0004_UUID@10.0.11.91@o2ib10:665/0 lens 224/224 e 0 to 0 dl 1547251610 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:10.0:1547251604.895529:0:13336:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821840 00000100:00000200:8.0:1547251604.895532:0:13370:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821824 00010000:00000200:10.0:1547251604.895534:0:13336:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b17b07a100 x1622383954821840/t0(0) o400->cache1-MDT0000-lwp-OST0003_UUID@10.0.11.91@o2ib10:665/0 lens 224/224 e 0 to 0 dl 1547251610 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:11.0:1547251604.895537:0:12983:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821856, offset 192 00000400:00000200:11.0:1547251604.895540:0:12983:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00010000:00000200:8.0:1547251604.895540:0:13370:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b17b079800 x1622383954821824/t0(0) o400->cache1-MDT0000-lwp-OST0005_UUID@10.0.11.91@o2ib10:665/0 lens 224/224 e 0 to 0 dl 1547251610 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:10.0:1547251604.895541:0:13336:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821840, offset 192 00000400:00000200:11.0:1547251604.895544:0:12983:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:10.0:1547251604.895545:0:13336:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000400:00000200:11.0:1547251604.895547:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251604.895549:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:8.0:1547251604.895549:0:13370:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821824, offset 192 00000400:00000200:11.0:1547251604.895551:0:12983:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9746, 9746] 00000400:00000200:11.0:1547251604.895553:0:12983:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:8.0:1547251604.895553:0:13370:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000400:00000200:11.0:1547251604.895560:0:12983:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251604.895564:0:12983:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:10.0:1547251604.895564:0:13336:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000400:00000200:10.0:1547251604.895567:0:13336:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251604.895568:0:13336:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251604.895570:0:13336:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9746, 9747] 00000400:00000200:10.0:1547251604.895572:0:13336:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251604.895578:0:13336:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.91@o2ib10(10.1.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:10.0:1547251604.895582:0:13336:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:8.0:1547251604.895599:0:13370:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251604.895602:0:13370:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251604.895604:0:13370:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251604.895606:0:13370:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9747, 9747] 00000400:00000200:8.0:1547251604.895609:0:13370:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:8.0:1547251604.895616:0:13370:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.1.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251604.895619:0:13370:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:8.0:1547251604.895672:0:13370:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251604.895673:0:12983:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:10.0:1547251604.895673:0:13336:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251604.895674:0:12983:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251604.895674:0:13336:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:8.0:1547251604.895674:0:13370:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:10.0:1547251604.895675:0:13336:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251604.895676:0:12983:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (22)++ 00000800:00000200:8.0:1547251604.895676:0:13370:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251604.895678:0:12983:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (23)-- 00000800:00000200:10.0:1547251604.895678:0:13336:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000800:00000200:8.0:1547251604.895679:0:13370:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:11.0:1547251604.895688:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 26 MB=0x5c38cb35102b0 00000800:00000200:5.2:1547251604.895690:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (22)++ 00000400:00000200:11.0:1547251604.895692:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0xd4f79 [1] + 0 00000400:00000200:11.0:1547251604.895695:0:12663:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d3b8 00000100:00000200:11.0:1547251604.895697:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:11.0:1547251604.895705:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d3b8 00000400:00000200:11.0:1547251604.895706:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251604.895708:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (23)++ 00000800:00000200:11.0:1547251604.895709:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (23)++ 00000800:00000200:11.0:1547251604.895710:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (25)-- 00000800:00000200:10.0:1547251604.895710:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (25)-- 00000800:00000200:11.0:1547251604.895711:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (23)-- 00000400:00000200:10.0:1547251604.895711:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bdd0 00000400:00000200:10.0:1547251604.895712:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251604.895713:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251604.895716:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251604.895718:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:7.2:1547251604.895730:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000100:00000200:0.0:1547251604.895730:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821808 00010000:00000200:0.0:1547251604.895736:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2df450 x1622383954821808/t0(0) o400->dc7951f7-3e71-e9c4-2e5e-5f95f4837d68@10.0.11.91@o2ib10:665/0 lens 224/224 e 0 to 0 dl 1547251610 ref 1 fl Interpret:/0/0 rc 0/0 00000800:00000200:7.2:1547251604.895740:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000100:00000200:0.0:1547251604.895744:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954821808, offset 192 00000800:00000200:11.0:1547251604.895747:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000400:00000200:0.0:1547251604.895747:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:11.0:1547251604.895750:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000400:00000200:0.0:1547251604.895751:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:11.0:1547251604.895752:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a9144c8 00000400:00000200:11.0:1547251604.895753:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251604.895753:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000400:00000200:0.0:1547251604.895753:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:10.0:1547251604.895754:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000400:00000200:0.0:1547251604.895755:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251604.895756:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251604.895756:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000400:00000200:10.0:1547251604.895756:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:0.0:1547251604.895756:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9747, 9748] 00000800:00000200:11.0:1547251604.895758:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:0.0:1547251604.895758:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251604.895760:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000400:00000200:10.0:1547251604.895760:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251604.895761:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:0.0:1547251604.895763:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:10.0:1547251604.895766:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:0.0:1547251604.895766:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:0.0:1547251604.895769:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251604.895770:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:0.0:1547251604.895772:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000800:00000200:0.0:1547251604.895774:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)-- 00000100:00000200:0.0:1547251604.895778:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:5.2:1547251604.895785:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251604.895797:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251604.895802:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000400:00000200:11.0:1547251604.895805:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665000 00000400:00000200:11.0:1547251604.895807:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251604.895808:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:11.0:1547251604.895810:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251604.895812:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:9.0:1547251605.066676:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444720, portal 4 00000100:00000200:9.0:1547251605.066679:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444720, offset 0 00000400:00000200:9.0:1547251605.066682:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:9.0:1547251605.066685:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:9.0:1547251605.066687:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23028] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251605.066690:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23027] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23028] 00000400:00000200:9.0:1547251605.066692:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251605.066695:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251605.066696:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251605.066697:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9742, 9742] 00000400:00000200:9.0:1547251605.066699:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:9.0:1547251605.066704:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251605.066706:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:9.0:1547251605.066709:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251605.066711:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:9.0:1547251605.066712:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:9.0:1547251605.066714:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:9.2:1547251605.066723:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000100:00000200:1.0:1547251605.066724:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444752, portal 4 00000100:00000200:1.0:1547251605.066728:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444752, offset 0 00000400:00000200:1.0:1547251605.066732:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:1.0:1547251605.066735:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:1.0:1547251605.066737:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23028] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251605.066740:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23028] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23028] 00000400:00000200:1.0:1547251605.066743:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:1.0:1547251605.066744:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251605.066746:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251605.066747:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9742, 9743] 00000800:00000200:14.0:1547251605.066748:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000400:00000200:1.0:1547251605.066749:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000100:00000200:3.0:1547251605.066753:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444736, portal 4 00000400:00000200:1.0:1547251605.066753:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:1.0:1547251605.066756:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000100:00000200:3.0:1547251605.066757:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444736, offset 0 00000800:00000200:14.0:1547251605.066759:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:1.0:1547251605.066759:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000400:00000200:3.0:1547251605.066760:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:1.0:1547251605.066760:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:1.0:1547251605.066761:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:14.0:1547251605.066763:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf7b00 x1622384012444720/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251612 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251605.066763:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000800:00000200:1.0:1547251605.066764:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:3.0:1547251605.066766:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23029] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:14.0:1547251605.066769:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea088 00000400:00000200:3.0:1547251605.066769:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23028] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23029] 00000400:00000200:14.0:1547251605.066770:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:3.0:1547251605.066771:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:14.0:1547251605.066773:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:3.0:1547251605.066773:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:14.0:1547251605.066775:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:3.0:1547251605.066775:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251605.066776:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9743, 9743] 00000800:00000200:14.0:1547251605.066777:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000400:00000200:3.0:1547251605.066778:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000800:00000200:7.2:1547251605.066779:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:3.0:1547251605.066783:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251605.066786:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:3.0:1547251605.066789:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251605.066791:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:3.0:1547251605.066792:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:3.0:1547251605.066795:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:14.0:1547251605.066796:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:14.0:1547251605.066798:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:14.0:1547251605.066800:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b7800 x1622384012444752/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251612 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251605.066804:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2bb0 00000800:00000200:12.0:1547251605.066804:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:14.0:1547251605.066805:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251605.066807:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:9.2:1547251605.066808:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:14.0:1547251605.066810:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:14.0:1547251605.066812:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:14.0:1547251605.066813:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:12.0:1547251605.066814:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:14.0:1547251605.066815:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328dd700 x1622384012444736/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251612 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251605.066818:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555660 00000400:00000200:14.0:1547251605.066819:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251605.066821:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251605.066822:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:9.2:1547251605.066862:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:12.0:1547251605.066874:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251605.066883:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251605.066885:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251605.066886:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:12.0:1547251605.066887:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:14.0:1547251605.066888:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:14.0:1547251605.066894:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000800:00000200:7.2:1547251605.066894:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:12.0:1547251605.066895:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:14.0:1547251605.066897:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251605.066897:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251605.066899:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251605.066901:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04430 00000400:00000200:12.0:1547251605.066903:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251605.066904:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fb1d [1] + 192 00000100:00000200:14.0:1547251605.066907:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf7b00 x1622384012444720/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251612 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251605.066908:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04450 00000800:00000200:7.2:1547251605.066911:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000400:00000200:14.0:1547251605.066915:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251605.066917:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:12.0:1547251605.066917:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fb2d [1] + 192 00000800:00000200:14.0:1547251605.066918:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:14.0:1547251605.066919:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:14.0:1547251605.066921:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:12.0:1547251605.066921:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b7800 x1622384012444752/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251612 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251605.066923:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:14.0:1547251605.066924:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251605.066927:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251605.066929:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04440 00000400:00000200:12.0:1547251605.066930:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251605.066932:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:9.0:1547251605.066932:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf7b00 x1622384012444720/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251612 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251605.066934:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251605.066934:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:12.0:1547251605.066936:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000100:00000200:9.0:1547251605.066938:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf7b00 x1622384012444720/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251612 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251605.066956:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea6e8 00000400:00000200:14.0:1547251605.066959:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fb3d [1] + 192 00010000:00080000:9.0:1547251605.066960:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf7b00 x1622384012444720/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251612 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:1.0:1547251605.066963:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b7800 x1622384012444752/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251612 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:1.0:1547251605.066971:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b7800 x1622384012444752/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251612 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251605.066977:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00000100:00000200:14.0:1547251605.066979:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328dd700 x1622384012444736/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251612 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:1.0:1547251605.066981:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b7800 x1622384012444752/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251612 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:14.0:1547251605.066986:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251605.066988:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (21)++ 00000800:00000200:14.0:1547251605.066989:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (22)-- 00000800:00000200:14.0:1547251605.066990:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:3.0:1547251605.066997:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328dd700 x1622384012444736/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251612 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251605.067002:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328dd700 x1622384012444736/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251612 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251605.067006:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251605.067010:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328dd700 x1622384012444736/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251612 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251606.522689:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012444816, portal 10 00000100:00000200:4.0:1547251606.522690:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012444784, portal 10 00000100:00000200:5.0:1547251606.522693:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012444816, offset 0 00000100:00000200:4.0:1547251606.522693:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012444784, offset 0 00000400:00000200:5.0:1547251606.522696:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251606.522700:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251606.522700:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251606.522703:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251606.522704:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c04490 00000400:00000200:4.0:1547251606.522706:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c04470 00000400:00000200:4.0:1547251606.522709:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 60480 00000400:00000200:5.0:1547251606.522712:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 60704 00000100:00000200:6.0:1547251606.522714:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012444768, portal 25 00000100:00000200:4.0:1547251606.522715:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:4.0:1547251606.522724:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251606.522725:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:5.0:1547251606.522728:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251606.522729:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 26, xid 1622384012444768, offset 0 00000100:00000200:4.0:1547251606.522730:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c434e00 x1622384012444784/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251606.522737:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b5d8 00000400:00000200:4.0:1547251606.522738:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:5.0:1547251606.522739:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c434200 x1622384012444816/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251606.522740:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:5.0:1547251606.522745:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adcc0 00000400:00000200:5.0:1547251606.522746:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251606.522748:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251606.522748:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251606.522751:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012444800, portal 10 00000400:00000200:6.0:1547251606.522752:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:4.0:1547251606.522754:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012444800, offset 0 00000400:00000200:6.0:1547251606.522756:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 26 MB=0x5c38cb6c04460 00000400:00000200:4.0:1547251606.522756:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251606.522759:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-0@lo of length 224/224 into md 0xd4f89 [1] + 0 00000400:00000200:4.0:1547251606.522759:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:7.0:1547251606.522761:0:12981:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444784 00000400:00000200:4.0:1547251606.522762:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c04480 00000400:00000200:6.0:1547251606.522763:0:12676:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52da18 00000100:00000200:6.0:1547251606.522765:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:4.0:1547251606.522765:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 60928 00010000:00000200:7.0:1547251606.522768:0:12981:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444e600 x1622384012444784/t0(0) o400->cache1-MDT0000-lwp-MDT0000_UUID@0@lo:667/0 lens 224/224 e 0 to 0 dl 1547251612 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:4.0:1547251606.522768:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:6.0:1547251606.522775:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52da18 00000400:00000200:6.0:1547251606.522776:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251606.522776:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251606.522777:0:12981:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012444784, offset 192 00000100:00000200:6.0:1547251606.522779:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c431800 x1622384012444768/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251613 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251606.522785:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990b28 00000400:00000200:6.0:1547251606.522786:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251606.522788:0:12981:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251606.522788:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251606.522790:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c436000 x1622384012444800/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251606.522791:0:12981:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:6.0:1547251606.522794:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012444832, portal 10 00000400:00000200:7.0:1547251606.522795:0:12981:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c04470 00000400:00000200:4.0:1547251606.522796:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b550 00000100:00000200:6.0:1547251606.522797:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012444832, offset 0 00000100:00000200:5.0:1547251606.522797:0:12980:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444816 00000400:00000200:4.0:1547251606.522797:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251606.522798:0:12981:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaee11 [1] + 192 00000400:00000200:4.0:1547251606.522798:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251606.522800:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:7.0:1547251606.522802:0:12981:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c434e00 x1622384012444784/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251606.522802:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00010000:00000200:5.0:1547251606.522802:0:12980:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae5c432100 x1622384012444816/t0(0) o400->cache1-MDT0000-lwp-OST0002_UUID@0@lo:667/0 lens 224/224 e 0 to 0 dl 1547251612 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:6.0:1547251606.522805:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c044a0 00000400:00000200:6.0:1547251606.522808:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 61152 00000400:00000200:7.0:1547251606.522810:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251606.522810:0:13338:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444800 00000100:00000200:6.0:1547251606.522811:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:5.0:1547251606.522811:0:12980:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012444816, offset 192 00000400:00000200:6.0:1547251606.522814:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00010000:00000200:4.0:1547251606.522814:0:13338:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444c800 x1622384012444800/t0(0) o400->cache1-MDT0000-lwp-OST0000_UUID@0@lo:667/0 lens 224/224 e 0 to 0 dl 1547251612 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:7.0:1547251606.522815:0:12981:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea908 00000400:00000200:7.0:1547251606.522816:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251606.522817:0:12980:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251606.522818:0:12981:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251606.522819:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c431500 x1622384012444832/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251606.522820:0:12980:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251606.522823:0:12980:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c04490 00000400:00000200:6.0:1547251606.522824:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990440 00000400:00000200:6.0:1547251606.522825:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251606.522826:0:12980:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaee01 [1] + 192 00000100:00000200:4.0:1547251606.522826:0:13338:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012444800, offset 192 00000400:00000200:6.0:1547251606.522827:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251606.522828:0:12981:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444832 00000400:00000200:4.0:1547251606.522828:0:13338:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:5.0:1547251606.522830:0:12980:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c434200 x1622384012444816/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251606.522831:0:13338:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00010000:00000200:7.0:1547251606.522832:0:12981:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a765df6300 x1622384012444832/t0(0) o400->cache1-MDT0000-lwp-OST0001_UUID@0@lo:667/0 lens 224/224 e 0 to 0 dl 1547251612 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:4.0:1547251606.522833:0:13338:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c04480 00000100:00000200:0.0:1547251606.522836:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444768 00000400:00000200:5.0:1547251606.522837:0:12980:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251606.522838:0:12981:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012444832, offset 192 00000400:00000200:4.0:1547251606.522838:0:13338:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaee31 [1] + 192 00000400:00000200:5.0:1547251606.522839:0:12980:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adcc0 00000400:00000200:5.0:1547251606.522840:0:12980:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251606.522841:0:12981:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:4.0:1547251606.522841:0:13338:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c436000 x1622384012444800/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00010000:00000200:0.0:1547251606.522841:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2dcc50 x1622384012444768/t0(0) o400->a65be517-33e7-fa36-0de3-da9e02df4f8d@0@lo:667/0 lens 224/224 e 0 to 0 dl 1547251612 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:5.0:1547251606.522842:0:12980:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251606.522844:0:12981:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251606.522847:0:12981:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c044a0 00000400:00000200:4.0:1547251606.522848:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:0.0:1547251606.522849:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384012444768, offset 192 00000400:00000200:7.0:1547251606.522850:0:12981:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaee41 [1] + 192 00000400:00000200:4.0:1547251606.522850:0:13338:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b550 00000400:00000200:4.0:1547251606.522851:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251606.522852:0:13338:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251606.522853:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c434200 x1622384012444816/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251606.522859:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c434200 x1622384012444816/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251606.522864:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad908 00000400:00000200:0.0:1547251606.522866:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:7.0:1547251606.522869:0:12981:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c431500 x1622384012444832/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251606.522869:0:12929:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:0.0:1547251606.522872:0:12929:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 25 MB=0x5c38cb6c04460 00000400:00000200:7.0:1547251606.522878:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251606.522879:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c434e00 x1622384012444784/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251606.522883:0:12981:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea908 00000400:00000200:7.0:1547251606.522884:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251606.522885:0:12981:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251606.522892:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c434e00 x1622384012444784/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251606.522898:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bdd0 00000400:00000200:0.0:1547251606.522900:0:12929:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 19 from 12345-0@lo of length 192/192 into md 0xaee21 [1] + 192 00000100:00000200:6.0:1547251606.522903:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c431500 x1622384012444832/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251606.522908:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c431500 x1622384012444832/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251606.522912:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990b28 00000100:00000200:0.0:1547251606.522914:0:12929:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c431800 x1622384012444768/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251606.522922:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251606.522923:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c436000 x1622384012444800/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251606.522927:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c436000 x1622384012444800/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251606.522930:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b5d8 00000400:00000200:0.0:1547251606.522931:0:12929:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665198 00000400:00000200:0.0:1547251606.522932:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:6.0:1547251606.522933:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c431800 x1622384012444768/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251606.522934:0:12929:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251606.522938:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c431800 x1622384012444768/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251613 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251606.522938:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000400:00000200:6.0:1547251606.522949:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459905d8 00000100:00000200:8.0:1547251607.178682:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444848, portal 4 00000100:00000200:8.0:1547251607.178685:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444848, offset 0 00000400:00000200:8.0:1547251607.178689:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251607.178691:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251607.178694:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23029] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251607.178697:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23029] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23029] 00000400:00000200:8.0:1547251607.178699:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251607.178701:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251607.178702:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251607.178703:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9748, 9748] 00000400:00000200:8.0:1547251607.178705:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:8.0:1547251607.178710:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251607.178712:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:8.0:1547251607.178715:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251607.178717:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:8.0:1547251607.178718:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:8.0:1547251607.178720:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000100:00000200:2.0:1547251607.178734:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444864, portal 4 00000100:00000200:2.0:1547251607.178737:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444864, offset 0 00000400:00000200:2.0:1547251607.178740:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:2.0:1547251607.178743:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:2.0:1547251607.178745:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23030] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251607.178748:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23029] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23030] 00000400:00000200:2.0:1547251607.178750:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251607.178751:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251607.178753:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251607.178754:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9748, 9749] 00000400:00000200:2.0:1547251607.178756:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:2.0:1547251607.178760:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251607.178763:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000100:00000200:12.0:1547251607.178766:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444880, portal 4 00000800:00000200:2.0:1547251607.178766:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251607.178767:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:2.0:1547251607.178768:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000100:00000200:12.0:1547251607.178769:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444880, offset 0 00000800:00000200:2.0:1547251607.178771:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000400:00000200:12.0:1547251607.178772:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:12.0:1547251607.178774:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:5.2:1547251607.178776:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000400:00000200:12.0:1547251607.178777:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23030] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251607.178780:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23030] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23030] 00000400:00000200:12.0:1547251607.178782:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:12.0:1547251607.178783:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251607.178785:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251607.178786:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9749, 9749] 00000400:00000200:12.0:1547251607.178788:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:12.0:1547251607.178793:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:10.0:1547251607.178793:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:12.0:1547251607.178795:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:12.0:1547251607.178799:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251607.178800:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (23)++ 00000800:00000200:12.0:1547251607.178801:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (24)++ 00000800:00000200:12.0:1547251607.178804:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (25)-- 00000800:00000200:10.0:1547251607.178805:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (24)-- 00000100:00000200:10.0:1547251607.178810:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824756300 x1622384012444848/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251607.178817:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000800:00000200:11.0:1547251607.178819:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (23)++ 00000400:00000200:10.0:1547251607.178819:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251607.178821:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (24)-- 00000800:00000200:7.2:1547251607.178821:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000400:00000200:10.0:1547251607.178822:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000100:00000200:11.0:1547251607.178825:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cad700 x1622384012444880/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251607.178825:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (23)-- 00000800:00000200:10.0:1547251607.178827:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251607.178830:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000400:00000200:11.0:1547251607.178831:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038aa0 00000400:00000200:11.0:1547251607.178832:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251607.178832:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000100:00000200:10.0:1547251607.178835:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f6600 x1622384012444864/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251607.178836:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251607.178838:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:11.0:1547251607.178840:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:10.0:1547251607.178841:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a990 00000400:00000200:10.0:1547251607.178842:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251607.178845:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251607.178847:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:5.2:1547251607.178891:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251607.178897:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:5.2:1547251607.178903:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:11.0:1547251607.178906:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251607.178911:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251607.178911:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251607.178913:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251607.178916:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251607.178918:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251607.178920:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c044b0 00000400:00000200:10.0:1547251607.178921:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c044d0 00000400:00000200:11.0:1547251607.178924:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6ce9 [1] + 192 00000400:00000200:10.0:1547251607.178929:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6d19 [1] + 192 00000100:00000200:11.0:1547251607.178933:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824756300 x1622384012444848/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251607.178953:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251607.178955:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000100:00000200:10.0:1547251607.178955:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cad700 x1622384012444880/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251607.178957:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251607.178959:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251607.178961:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251607.178963:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251607.178965:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:11.0:1547251607.178967:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251607.178967:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000100:00000200:8.0:1547251607.178967:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824756300 x1622384012444848/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251607.178968:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251607.178969:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:11.0:1547251607.178971:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c044c0 00000800:00000200:10.0:1547251607.178971:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:8.0:1547251607.178972:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824756300 x1622384012444848/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251607.178973:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:10.0:1547251607.178974:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:8.0:1547251607.178977:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6883b8 00010000:00080000:8.0:1547251607.178980:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824756300 x1622384012444848/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:11.0:1547251607.178981:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6d09 [1] + 192 00000100:00000200:12.0:1547251607.178986:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cad700 x1622384012444880/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251607.178992:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cad700 x1622384012444880/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251607.178996:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b0383b8 00000100:00000200:11.0:1547251607.179000:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f6600 x1622384012444864/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:12.0:1547251607.179001:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cad700 x1622384012444880/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:11.0:1547251607.179009:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251607.179010:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251607.179012:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251607.179012:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:2.0:1547251607.179019:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f6600 x1622384012444864/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251607.179025:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f6600 x1622384012444864/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251607.179029:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00010000:00080000:2.0:1547251607.179033:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f6600 x1622384012444864/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251607.642696:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444896, portal 4 00000100:00000200:9.0:1547251607.642699:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444896, offset 0 00000100:00000200:2.0:1547251607.642701:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444928, portal 4 00000400:00000200:9.0:1547251607.642702:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251607.642704:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000100:00000200:2.0:1547251607.642705:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444928, offset 0 00000400:00000200:9.0:1547251607.642707:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23031] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251607.642708:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251607.642710:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23030] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23031] 00000400:00000200:2.0:1547251607.642711:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251607.642713:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251607.642713:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23031] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251607.642714:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251607.642716:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251607.642716:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23031] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23031] 00000400:00000200:9.0:1547251607.642717:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9741, 9742] 00000400:00000200:2.0:1547251607.642718:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251607.642719:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:2.0:1547251607.642720:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251607.642722:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251607.642723:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9742, 9742] 00000400:00000200:9.0:1547251607.642724:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000100:00000200:13.0:1547251607.642725:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444912, portal 4 00000400:00000200:2.0:1547251607.642725:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:9.0:1547251607.642726:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000100:00000200:13.0:1547251607.642729:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444912, offset 0 00000800:00000200:9.0:1547251607.642729:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:2.0:1547251607.642730:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251607.642732:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000400:00000200:13.0:1547251607.642733:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:9.0:1547251607.642733:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:2.0:1547251607.642733:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:9.0:1547251607.642735:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:13.0:1547251607.642736:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:2.0:1547251607.642736:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251607.642738:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000400:00000200:13.0:1547251607.642739:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23032] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:2.0:1547251607.642739:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:2.0:1547251607.642741:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000400:00000200:13.0:1547251607.642742:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23031] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23032] 00000400:00000200:13.0:1547251607.642744:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:13.0:1547251607.642746:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251607.642748:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251607.642749:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9742, 9743] 00000400:00000200:13.0:1547251607.642752:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:0.2:1547251607.642752:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:13.0:1547251607.642760:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:12.2:1547251607.642760:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:13.0:1547251607.642764:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:13.0:1547251607.642768:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251607.642770:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:13.0:1547251607.642771:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (23)++ 00000800:00000200:13.0:1547251607.642775:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:4.0:1547251607.642812:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)++ 00000800:00000200:5.0:1547251607.642816:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251607.642816:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:5.0:1547251607.642818:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000100:00000200:4.0:1547251607.642820:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf7b00 x1622384012444896/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251607.642822:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f6300 x1622384012444928/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251607.642828:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06acc0 00000400:00000200:4.0:1547251607.642828:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea770 00000400:00000200:4.0:1547251607.642829:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251607.642831:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251607.642833:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:5.0:1547251607.642834:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251607.642836:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251607.642837:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251607.642838:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251607.642839:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:4.0:1547251607.642839:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251607.642841:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:4.0:1547251607.642842:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a3c00 x1622384012444912/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251607.642848:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b990 00000400:00000200:4.0:1547251607.642849:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251607.642852:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251607.642854:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:0.2:1547251607.642877:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:5.0:1547251607.642891:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251607.642900:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.1.11.92@o2ib10 00000800:00000200:5.0:1547251607.642902:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251607.642904:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251607.642905:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251607.642906:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:0.2:1547251607.642916:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:12.2:1547251607.642919:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:5.0:1547251607.642927:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251607.642930:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000800:00000200:4.0:1547251607.642933:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000400:00000200:5.0:1547251607.642935:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251607.642935:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251607.642949:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251607.642953:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c044e0 00000400:00000200:4.0:1547251607.642957:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3b15 [1] + 192 00000400:00000200:5.0:1547251607.642959:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04500 00000400:00000200:5.0:1547251607.642962:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3b25 [1] + 192 00000100:00000200:4.0:1547251607.642967:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf7b00 x1622384012444896/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251607.642984:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251607.642986:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f6300 x1622384012444928/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251607.642987:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251607.642989:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:4.0:1547251607.642990:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251607.642992:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251607.642994:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000400:00000200:5.0:1547251607.642996:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251607.642996:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000100:00000200:9.0:1547251607.642998:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf7b00 x1622384012444896/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251607.642998:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251607.643000:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251607.643001:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000400:00000200:4.0:1547251607.643001:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:5.0:1547251607.643003:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000100:00000200:9.0:1547251607.643004:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf7b00 x1622384012444896/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251607.643004:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c044f0 00000400:00000200:9.0:1547251607.643008:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea6e8 00010000:00080000:9.0:1547251607.643011:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf7b00 x1622384012444896/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:2.0:1547251607.643011:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f6300 x1622384012444928/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251607.643017:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f6300 x1622384012444928/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251607.643021:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00000400:00000200:4.0:1547251607.643024:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3b35 [1] + 192 00010000:00080000:2.0:1547251607.643025:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f6300 x1622384012444928/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251607.643028:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a3c00 x1622384012444912/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251607.643035:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251607.643037:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251607.643038:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251607.643038:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000100:00000200:13.0:1547251607.643046:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a3c00 x1622384012444912/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:13.0:1547251607.643052:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a3c00 x1622384012444912/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251607.643056:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b3b8 00010000:00080000:13.0:1547251607.643060:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a3c00 x1622384012444912/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251614 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251608.042676:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444944, portal 4 00000100:00000200:9.0:1547251608.042679:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444944, offset 0 00000400:00000200:9.0:1547251608.042682:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251608.042684:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251608.042687:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04510 00000400:00000200:9.0:1547251608.042689:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd45f9 [1] + 0 00000400:00000200:9.0:1547251608.042692:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ae6a914b28 00000100:00000200:9.0:1547251608.042694:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251608.042702:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914b28 00000400:00000200:9.0:1547251608.042703:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251608.042707:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf7500 x1622384012444944/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251615 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251608.042713:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea990 00000400:00000200:9.0:1547251608.042714:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251608.042716:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251608.042724:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444944 00010000:00000200:8.0:1547251608.042734:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3db850 x1622384012444944/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:669/0 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251608.042738:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444960, portal 4 00000100:00000200:8.0:1547251608.042741:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444944, offset 192 00000100:00000200:6.0:1547251608.042741:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444960, offset 0 00000400:00000200:8.0:1547251608.042743:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251608.042745:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251608.042747:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251608.042749:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04510 00000400:00000200:6.0:1547251608.042749:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251608.042751:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04520 00000400:00000200:8.0:1547251608.042752:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaee81 [1] + 192 00000100:00000200:8.0:1547251608.042754:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf7500 x1622384012444944/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251615 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251608.042754:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe22e5 [1] + 0 00000400:00000200:6.0:1547251608.042756:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040bee0 00000100:00000200:6.0:1547251608.042758:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251608.042763:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251608.042765:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6883b8 00000400:00000200:8.0:1547251608.042766:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251608.042768:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251608.042768:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bee0 00000400:00000200:6.0:1547251608.042769:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251608.042771:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b82444a700 x1622384012444960/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251615 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251608.042773:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf7500 x1622384012444944/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251615 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251608.042775:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:6.0:1547251608.042776:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990b28 00000400:00000200:6.0:1547251608.042777:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251608.042778:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251608.042779:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf7500 x1622384012444944/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251615 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251608.042784:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea6e8 00010000:00080000:9.0:1547251608.042788:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf7500 x1622384012444944/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251615 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251608.042789:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444960 00010000:00000200:4.0:1547251608.042801:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39cc50 x1622384012444960/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:669/0 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251608.042810:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444960, offset 192 00000400:00000200:4.0:1547251608.042813:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251608.042817:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251608.042820:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04520 00000400:00000200:4.0:1547251608.042823:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaee91 [1] + 192 00000100:00000200:4.0:1547251608.042825:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b82444a700 x1622384012444960/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251615 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251608.042835:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251608.042837:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b5d8 00000400:00000200:4.0:1547251608.042838:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251608.042839:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251608.042844:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b82444a700 x1622384012444960/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251615 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251608.042844:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251608.042849:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b82444a700 x1622384012444960/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251615 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251608.042854:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459905d8 00010000:00080000:6.0:1547251608.042857:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b82444a700 x1622384012444960/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251615 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251608.058682:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444976, portal 4 00000100:00000200:4.0:1547251608.058685:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444976, offset 0 00000400:00000200:4.0:1547251608.058687:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251608.058690:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251608.058693:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04530 00000400:00000200:4.0:1547251608.058695:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2315 [1] + 0 00000400:00000200:4.0:1547251608.058697:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040b088 00000100:00000200:4.0:1547251608.058699:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:4.0:1547251608.058707:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b088 00000400:00000200:4.0:1547251608.058709:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251608.058713:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c434200 x1622384012444976/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251615 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251608.058719:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b550 00000400:00000200:4.0:1547251608.058720:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251608.058722:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251608.058731:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012444976 00010000:00000200:5.0:1547251608.058741:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39c850 x1622384012444976/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:669/0 lens 224/368 e 0 to 0 dl 1547251614 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251608.058748:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012444976, offset 192 00000400:00000200:5.0:1547251608.058750:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251608.058752:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251608.058755:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04530 00000400:00000200:5.0:1547251608.058757:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaeea1 [1] + 192 00000100:00000200:5.0:1547251608.058760:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c434200 x1622384012444976/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251615 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251608.058769:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251608.058773:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad908 00000400:00000200:5.0:1547251608.058774:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251608.058776:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251608.058778:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c434200 x1622384012444976/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251615 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251608.058783:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:4.0:1547251608.058784:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c434200 x1622384012444976/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251615 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251608.058790:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bdd0 00010000:00080000:4.0:1547251608.058794:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae5c434200 x1622384012444976/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251615 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251610.074676:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012444992, portal 4 00000100:00000200:8.0:1547251610.074679:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012444992, offset 0 00000400:00000200:8.0:1547251610.074682:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:8.0:1547251610.074684:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:8.0:1547251610.074687:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23032] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251610.074690:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23032] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23032] 00000400:00000200:8.0:1547251610.074692:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251610.074694:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251610.074695:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251610.074696:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9743, 9744] 00000400:00000200:8.0:1547251610.074698:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:8.0:1547251610.074703:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000100:00000200:0.0:1547251610.074704:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445008, portal 4 00000800:00000200:8.0:1547251610.074705:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000100:00000200:0.0:1547251610.074707:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445008, offset 0 00000800:00000200:8.0:1547251610.074708:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251610.074710:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:8.0:1547251610.074711:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:0.0:1547251610.074711:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:8.0:1547251610.074713:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:0.0:1547251610.074714:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:0.0:1547251610.074716:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23033] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251610.074719:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23032] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23033] 00000400:00000200:0.0:1547251610.074722:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251610.074723:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251610.074725:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251610.074726:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9744, 9744] 00000400:00000200:0.0:1547251610.074728:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:0.0:1547251610.074732:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251610.074735:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:0.0:1547251610.074738:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251610.074739:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:0.0:1547251610.074740:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:0.0:1547251610.074743:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:9.2:1547251610.074758:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:7.2:1547251610.074765:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:2.0:1547251610.074766:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445024, portal 4 00000100:00000200:2.0:1547251610.074770:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445024, offset 0 00000400:00000200:2.0:1547251610.074774:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:2.0:1547251610.074777:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:2.0:1547251610.074780:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23033] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251610.074784:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23033] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23033] 00000400:00000200:2.0:1547251610.074786:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:2.0:1547251610.074787:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251610.074789:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251610.074790:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9744, 9745] 00000400:00000200:2.0:1547251610.074792:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:2.0:1547251610.074796:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251610.074799:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:2.0:1547251610.074802:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251610.074803:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:2.0:1547251610.074804:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (23)++ 00000800:00000200:2.0:1547251610.074807:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (24)-- 00000800:00000200:12.0:1547251610.074811:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251610.074814:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:12.0:1547251610.074818:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328dda00 x1622384012445008/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251617 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251610.074823:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665dd0 00000400:00000200:12.0:1547251610.074824:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251610.074827:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251610.074830:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)++ 00000800:00000200:14.0:1547251610.074833:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (24)-- 00000800:00000200:12.0:1547251610.074833:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251610.074835:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:14.0:1547251610.074836:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824757800 x1622384012444992/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251617 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251610.074837:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)++ 00000800:00000200:12.0:1547251610.074838:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (24)-- 00000100:00000200:12.0:1547251610.074840:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f6000 x1622384012445024/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251617 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251610.074842:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000400:00000200:14.0:1547251610.074843:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251610.074844:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06add0 00000400:00000200:12.0:1547251610.074845:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251610.074846:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:12.0:1547251610.074847:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251610.074848:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251610.074849:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:14.0:1547251610.074850:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:9.2:1547251610.074896:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.2:1547251610.074901:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:9.2:1547251610.074907:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:12.0:1547251610.074909:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251610.074911:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000800:00000200:14.0:1547251610.074914:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:12.0:1547251610.074915:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:14.0:1547251610.074917:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251610.074919:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04550 00000400:00000200:14.0:1547251610.074920:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251610.074922:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fb7d [1] + 192 00000400:00000200:14.0:1547251610.074924:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04540 00000100:00000200:12.0:1547251610.074925:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328dda00 x1622384012445008/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251617 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251610.074936:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:14.0:1547251610.074937:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fb6d [1] + 192 00000800:00000200:12.0:1547251610.074938:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251610.074939:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:12.0:1547251610.074947:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:14.0:1547251610.074948:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824757800 x1622384012444992/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251617 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251610.074949:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251610.074950:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251610.074953:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251610.074957:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04560 00000400:00000200:14.0:1547251610.074959:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251610.074961:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000100:00000200:0.0:1547251610.074961:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328dda00 x1622384012445008/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251617 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251610.074962:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:14.0:1547251610.074963:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:14.0:1547251610.074965:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:14.0:1547251610.074966:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000100:00000200:0.0:1547251610.074967:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328dda00 x1622384012445008/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251617 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251610.074968:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:0.0:1547251610.074971:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665d48 00000400:00000200:12.0:1547251610.074974:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fb8d [1] + 192 00010000:00080000:0.0:1547251610.074975:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328dda00 x1622384012445008/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251617 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251610.074977:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824757800 x1622384012444992/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251617 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251610.074983:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824757800 x1622384012444992/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251617 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251610.074987:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00000100:00000200:12.0:1547251610.074990:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f6000 x1622384012445024/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251617 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251610.074992:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824757800 x1622384012444992/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251617 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:12.0:1547251610.074997:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251610.074999:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251610.075001:0:12665:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251610.075003:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251610.075003:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:2.0:1547251610.075007:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f6000 x1622384012445024/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251617 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251610.075013:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f6000 x1622384012445024/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251617 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:9.2:1547251610.075015:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:2.0:1547251610.075017:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00010000:00080000:2.0:1547251610.075020:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f6000 x1622384012445024/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251617 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:14.0:1547251610.075028:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:14.0:1547251610.075034:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:14.0:1547251610.075035:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:14.0:1547251610.075036:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000100:00000200:9.0:1547251612.186685:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445040, portal 4 00000100:00000200:9.0:1547251612.186688:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445040, offset 0 00000400:00000200:9.0:1547251612.186691:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251612.186694:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251612.186696:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23034] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251612.186699:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23033] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23034] 00000400:00000200:9.0:1547251612.186701:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251612.186702:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251612.186704:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251612.186705:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9749, 9750] 00000400:00000200:9.0:1547251612.186707:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:9.0:1547251612.186711:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251612.186714:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:9.0:1547251612.186717:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251612.186718:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:9.0:1547251612.186719:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:9.0:1547251612.186722:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000100:00000200:3.0:1547251612.186725:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445056, portal 4 00000100:00000200:3.0:1547251612.186728:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445056, offset 0 00000400:00000200:3.0:1547251612.186731:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:3.0:1547251612.186734:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:3.0:1547251612.186736:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23034] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251612.186739:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23034] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23034] 00000400:00000200:3.0:1547251612.186741:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251612.186743:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251612.186744:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251612.186746:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9750, 9750] 00000400:00000200:3.0:1547251612.186747:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000100:00000200:12.0:1547251612.186748:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445072, portal 4 00000100:00000200:12.0:1547251612.186752:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445072, offset 0 00000400:00000200:3.0:1547251612.186752:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251612.186755:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:12.0:1547251612.186756:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:3.0:1547251612.186758:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000400:00000200:12.0:1547251612.186760:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:3.0:1547251612.186760:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:3.0:1547251612.186761:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000400:00000200:12.0:1547251612.186763:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23035] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:3.0:1547251612.186764:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000400:00000200:12.0:1547251612.186768:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23034] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23035] 00000400:00000200:12.0:1547251612.186770:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:7.2:1547251612.186771:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000400:00000200:12.0:1547251612.186772:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251612.186773:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251612.186774:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9750, 9751] 00000400:00000200:12.0:1547251612.186776:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:12.0:1547251612.186781:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251612.186784:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:12.0:1547251612.186787:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:10.0:1547251612.186787:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:12.0:1547251612.186788:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (22)++ 00000800:00000200:12.0:1547251612.186789:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (24)++ 00000800:00000200:12.0:1547251612.186792:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (25)-- 00000800:00000200:10.0:1547251612.186797:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000100:00000200:10.0:1547251612.186801:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf7200 x1622384012445040/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251612.186810:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea990 00000400:00000200:10.0:1547251612.186811:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251612.186812:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:11.0:1547251612.186815:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000400:00000200:10.0:1547251612.186815:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:5.2:1547251612.186815:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000100:00000200:11.0:1547251612.186818:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8ca9500 x1622384012445072/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251612.186818:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251612.186820:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251612.186822:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000400:00000200:11.0:1547251612.186824:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038e58 00000800:00000200:10.0:1547251612.186824:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000400:00000200:11.0:1547251612.186826:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:10.0:1547251612.186828:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c3f00 x1622384012445056/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251612.186829:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251612.186831:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:11.0:1547251612.186833:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251612.186833:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245555d8 00000400:00000200:10.0:1547251612.186834:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251612.186837:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251612.186839:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:7.2:1547251612.186867:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:11.0:1547251612.186879:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251612.186887:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251612.186891:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251612.186892:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251612.186892:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251612.186893:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:7.2:1547251612.186904:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:5.2:1547251612.186908:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251612.186914:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:10.0:1547251612.186917:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251612.186920:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251612.186921:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251612.186924:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251612.186925:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251612.186928:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251612.186930:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04590 00000400:00000200:11.0:1547251612.186932:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04570 00000400:00000200:10.0:1547251612.186934:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6da9 [1] + 192 00000400:00000200:11.0:1547251612.186939:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6d79 [1] + 192 00000100:00000200:10.0:1547251612.186952:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8ca9500 x1622384012445072/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251612.186962:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251612.186964:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf7200 x1622384012445040/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251612.186964:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251612.186966:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251612.186967:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251612.186969:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251612.186971:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251612.186974:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:12.0:1547251612.186976:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8ca9500 x1622384012445072/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251612.186976:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000400:00000200:10.0:1547251612.186976:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251612.186977:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251612.186979:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:10.0:1547251612.186979:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04580 00000800:00000200:11.0:1547251612.186980:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:12.0:1547251612.186982:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8ca9500 x1622384012445072/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251612.186982:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:11.0:1547251612.186983:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:12.0:1547251612.186986:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b0383b8 00010000:00080000:12.0:1547251612.186990:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8ca9500 x1622384012445072/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251612.186990:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6d99 [1] + 192 00000100:00000200:9.0:1547251612.186995:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf7200 x1622384012445040/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251612.187000:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf7200 x1622384012445040/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251612.187005:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea6e8 00000100:00000200:10.0:1547251612.187007:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c3f00 x1622384012445056/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251612.187008:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf7200 x1622384012445040/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251612.187017:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251612.187018:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251612.187019:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251612.187020:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:3.0:1547251612.187026:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c3f00 x1622384012445056/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251612.187032:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c3f00 x1622384012445056/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251612.187036:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251612.187040:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c3f00 x1622384012445056/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251612.650679:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445088, portal 4 00000100:00000200:8.0:1547251612.650682:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445088, offset 0 00000400:00000200:8.0:1547251612.650685:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251612.650687:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251612.650689:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23035] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251612.650693:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23035] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23035] 00000400:00000200:8.0:1547251612.650695:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251612.650696:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251612.650698:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251612.650699:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9743, 9743] 00000400:00000200:8.0:1547251612.650701:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:8.0:1547251612.650705:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251612.650708:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:8.0:1547251612.650711:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251612.650712:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:8.0:1547251612.650713:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000100:00000200:12.0:1547251612.650714:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445120, portal 4 00000800:00000200:8.0:1547251612.650716:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000100:00000200:12.0:1547251612.650717:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445120, offset 0 00000400:00000200:12.0:1547251612.650720:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:12.2:1547251612.650724:0:12684:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000100:00000200:0.0:1547251612.650729:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445104, portal 4 00000100:00000200:0.0:1547251612.650733:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445104, offset 0 00000400:00000200:12.0:1547251612.650736:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:0.0:1547251612.650737:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:12.0:1547251612.650738:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23036] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251612.650740:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:12.0:1547251612.650741:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23035] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23036] 00000400:00000200:0.0:1547251612.650742:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23036] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251612.650743:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:12.0:1547251612.650745:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251612.650746:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23036] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23036] 00000400:00000200:12.0:1547251612.650747:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251612.650748:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9743, 9744] 00000400:00000200:0.0:1547251612.650748:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251612.650749:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251612.650750:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:0.0:1547251612.650752:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251612.650754:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9744, 9744] 00000400:00000200:12.0:1547251612.650755:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:0.0:1547251612.650756:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:12.0:1547251612.650758:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:0.0:1547251612.650760:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251612.650761:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251612.650762:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.0:1547251612.650763:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:0.0:1547251612.650763:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:12.0:1547251612.650766:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:0.0:1547251612.650766:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251612.650768:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:0.0:1547251612.650769:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:0.2:1547251612.650774:0:12670:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251612.650784:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:0.0:1547251612.650785:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251612.650788:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:0.2:1547251612.650792:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000100:00000200:5.0:1547251612.650794:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824754200 x1622384012445088/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251612.650801:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000800:00000200:4.0:1547251612.650801:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000400:00000200:5.0:1547251612.650803:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251612.650803:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:5.0:1547251612.650807:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000100:00000200:4.0:1547251612.650807:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f1800 x1622384012445120/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251612.650810:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251612.650812:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:5.0:1547251612.650813:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000400:00000200:4.0:1547251612.650813:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038e58 00000400:00000200:4.0:1547251612.650814:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:5.0:1547251612.650815:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000400:00000200:4.0:1547251612.650817:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000100:00000200:5.0:1547251612.650818:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c3c00 x1622384012445104/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251612.650820:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251612.650821:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:4.0:1547251612.650823:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:5.0:1547251612.650824:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00000400:00000200:5.0:1547251612.650825:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251612.650828:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251612.650829:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:12.2:1547251612.650848:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:4.0:1547251612.650861:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251612.650871:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.0.11.92@o2ib10 00000800:00000200:4.0:1547251612.650874:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251612.650875:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251612.650876:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:4.0:1547251612.650877:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251612.650879:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000800:00000200:4.0:1547251612.650879:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:5.0:1547251612.650883:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:0.2:1547251612.650883:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000400:00000200:5.0:1547251612.650888:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c045a0 00000800:00000200:4.0:1547251612.650891:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000400:00000200:5.0:1547251612.650892:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3ba5 [1] + 192 00000800:00000200:0.2:1547251612.650892:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:4.0:1547251612.650893:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000100:00000200:5.0:1547251612.650897:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824754200 x1622384012445088/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251612.650898:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251612.650902:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c045b0 00000400:00000200:5.0:1547251612.650909:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251612.650911:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3bc5 [1] + 192 00000800:00000200:5.0:1547251612.650912:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251612.650913:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251612.650914:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:4.0:1547251612.650915:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c3c00 x1622384012445104/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251612.650916:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251612.650920:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251612.650922:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251612.650926:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251612.650926:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251612.650927:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000100:00000200:8.0:1547251612.650928:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824754200 x1622384012445088/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251612.650929:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c045c0 00000800:00000200:4.0:1547251612.650929:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251612.650930:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251612.650932:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:8.0:1547251612.650933:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824754200 x1622384012445088/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251612.650937:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00010000:00080000:8.0:1547251612.650948:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824754200 x1622384012445088/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251612.650961:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3bb5 [1] + 192 00000100:00000200:0.0:1547251612.650965:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c3c00 x1622384012445104/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251612.650970:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c3c00 x1622384012445104/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251612.650975:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665d48 00000100:00000200:5.0:1547251612.650977:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f1800 x1622384012445120/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251612.650978:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c3c00 x1622384012445104/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251612.650985:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251612.650986:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251612.650988:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251612.650989:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:12.0:1547251612.650994:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f1800 x1622384012445120/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251612.650999:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f1800 x1622384012445120/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251612.651004:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b0383b8 00010000:00080000:12.0:1547251612.651007:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f1800 x1622384012445120/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251619 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251613.050695:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445136, portal 4 00000100:00000200:8.0:1547251613.050698:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445136, offset 0 00000400:00000200:8.0:1547251613.050701:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251613.050703:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251613.050706:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c045d0 00000400:00000200:8.0:1547251613.050709:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4679 [1] + 0 00000400:00000200:8.0:1547251613.050712:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ba9d367880 00000100:00000200:8.0:1547251613.050713:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251613.050722:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba9d367880 00000400:00000200:8.0:1547251613.050723:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251613.050727:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824756900 x1622384012445136/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251620 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251613.050733:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000400:00000200:8.0:1547251613.050734:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251613.050736:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251613.050744:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445136 00000100:00000200:4.0:1547251613.050748:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445152, portal 4 00000100:00000200:4.0:1547251613.050752:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445152, offset 0 00010000:00000200:9.0:1547251613.050754:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3db450 x1622384012445136/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:674/0 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:4.0:1547251613.050755:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251613.050759:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:9.0:1547251613.050761:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012445136, offset 192 00000400:00000200:4.0:1547251613.050762:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c045e0 00000400:00000200:9.0:1547251613.050763:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251613.050764:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2385 [1] + 0 00000400:00000200:4.0:1547251613.050766:0:12675:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad198 00000400:00000200:9.0:1547251613.050767:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:4.0:1547251613.050768:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251613.050770:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c045d0 00000400:00000200:9.0:1547251613.050772:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaeef1 [1] + 192 00000100:00000200:9.0:1547251613.050775:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824756900 x1622384012445136/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251620 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251613.050777:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad198 00000400:00000200:4.0:1547251613.050778:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251613.050782:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df1200 x1622384012445152/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251620 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251613.050784:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251613.050786:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea6e8 00000400:00000200:9.0:1547251613.050787:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251613.050788:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b550 00000400:00000200:9.0:1547251613.050789:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251613.050789:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251613.050791:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251613.050793:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824756900 x1622384012445136/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251620 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251613.050796:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:8.0:1547251613.050799:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824756900 x1622384012445136/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251620 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251613.050802:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445152 00000400:00000200:8.0:1547251613.050805:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00010000:00080000:8.0:1547251613.050809:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824756900 x1622384012445136/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251620 ref 1 fl Rpc:R/0/0 rc 0/0 00010000:00000200:5.0:1547251613.050812:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39c450 x1622384012445152/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:674/0 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251613.050819:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012445152, offset 192 00000400:00000200:5.0:1547251613.050821:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251613.050824:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251613.050826:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c045e0 00000400:00000200:5.0:1547251613.050829:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaef01 [1] + 192 00000100:00000200:5.0:1547251613.050832:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df1200 x1622384012445152/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251620 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251613.050842:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251613.050844:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adcc0 00000400:00000200:5.0:1547251613.050845:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251613.050847:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251613.050852:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df1200 x1622384012445152/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251620 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251613.050854:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:4.0:1547251613.050859:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df1200 x1622384012445152/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251620 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251613.050865:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bdd0 00010000:00080000:4.0:1547251613.050868:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df1200 x1622384012445152/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251620 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251613.066717:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445168, portal 4 00000100:00000200:7.0:1547251613.066721:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445168, offset 0 00000400:00000200:7.0:1547251613.066725:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251613.066728:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251613.066731:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c045f0 00000400:00000200:7.0:1547251613.066734:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe23b5 [1] + 0 00000400:00000200:7.0:1547251613.066736:0:12676:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad2a8 00000100:00000200:7.0:1547251613.066737:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:7.0:1547251613.066746:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad2a8 00000400:00000200:7.0:1547251613.066747:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251613.066749:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b82444e900 x1622384012445168/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251620 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251613.066754:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea2a8 00000400:00000200:7.0:1547251613.066755:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251613.066756:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251613.066777:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445168 00010000:00000200:5.0:1547251613.066785:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39c050 x1622384012445168/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:674/0 lens 224/368 e 0 to 0 dl 1547251619 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251613.066792:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012445168, offset 192 00000400:00000200:5.0:1547251613.066794:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251613.066797:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251613.066800:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c045f0 00000400:00000200:5.0:1547251613.066804:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaef11 [1] + 192 00000100:00000200:5.0:1547251613.066807:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b82444e900 x1622384012445168/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251620 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251613.066816:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251613.066818:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ada18 00000400:00000200:5.0:1547251613.066819:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251613.066820:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251613.066824:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b82444e900 x1622384012445168/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251620 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251613.066825:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251613.066829:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b82444e900 x1622384012445168/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251620 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251613.066833:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea908 00010000:00080000:7.0:1547251613.066836:0:12676:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b82444e900 x1622384012445168/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251620 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:2.0:1547251615.082689:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445200, portal 4 00000100:00000200:2.0:1547251615.082692:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445200, offset 0 00000400:00000200:2.0:1547251615.082695:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:2.0:1547251615.082697:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000100:00000200:9.0:1547251615.082699:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445184, portal 4 00000400:00000200:2.0:1547251615.082700:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23036] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:9.0:1547251615.082701:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445184, offset 0 00000400:00000200:2.0:1547251615.082703:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23037] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23036] 00000400:00000200:9.0:1547251615.082704:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:2.0:1547251615.082706:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251615.082707:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:2.0:1547251615.082709:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251615.082710:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23037] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251615.082711:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251615.082712:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9745, 9745] 00000400:00000200:9.0:1547251615.082713:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23037] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23037] 00000400:00000200:9.0:1547251615.082715:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251615.082717:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251615.082717:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:9.0:1547251615.082718:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251615.082720:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9745, 9746] 00000100:00000200:1.0:1547251615.082721:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445216, portal 4 00000400:00000200:9.0:1547251615.082722:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:2.0:1547251615.082724:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000100:00000200:1.0:1547251615.082725:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445216, offset 0 00000800:00000200:2.0:1547251615.082728:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:1.0:1547251615.082728:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:9.0:1547251615.082731:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:1.0:1547251615.082731:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:1.0:1547251615.082733:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:254, d:10, s:23038] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:9.0:1547251615.082734:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:1.0:1547251615.082736:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23037] with best_ni 10.0.11.90@o2ib10 [c:254, d:10, s:23038] 00000800:00000200:9.0:1547251615.082737:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251615.082738:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:9.0:1547251615.082739:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:1.0:1547251615.082739:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:1.0:1547251615.082740:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251615.082742:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:1.0:1547251615.082743:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251615.082744:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9746, 9746] 00000400:00000200:1.0:1547251615.082746:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000800:00000200:2.0:1547251615.082750:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251615.082751:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000400:00000200:1.0:1547251615.082751:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251615.082752:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:1.0:1547251615.082754:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:2.0:1547251615.082755:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:1.0:1547251615.082757:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251615.082758:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:1.0:1547251615.082759:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:1.0:1547251615.082762:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:9.2:1547251615.082777:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:7.2:1547251615.082791:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:7.2:1547251615.082801:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251615.082809:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251615.082812:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:12.0:1547251615.082815:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328ddd00 x1622384012445216/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251622 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251615.082821:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2bb0 00000400:00000200:12.0:1547251615.082823:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251615.082825:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251615.082827:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251615.082829:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251615.082830:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:14.0:1547251615.082831:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000100:00000200:12.0:1547251615.082832:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf6f00 x1622384012445184/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251622 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251615.082834:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000400:00000200:12.0:1547251615.082837:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaee0 00000400:00000200:12.0:1547251615.082837:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:14.0:1547251615.082838:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c3900 x1622384012445200/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251622 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251615.082839:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251615.082841:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251615.082842:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000400:00000200:14.0:1547251615.082843:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06add0 00000800:00000200:12.0:1547251615.082843:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000400:00000200:14.0:1547251615.082844:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:12.0:1547251615.082845:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000400:00000200:14.0:1547251615.082847:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251615.082851:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:7.2:1547251615.082912:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251615.082917:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:12.0:1547251615.082924:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251615.082927:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000800:00000200:14.0:1547251615.082930:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:12.0:1547251615.082930:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:14.0:1547251615.082932:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251615.082934:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04610 00000400:00000200:14.0:1547251615.082936:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251615.082937:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fbbd [1] + 192 00000400:00000200:14.0:1547251615.082940:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04600 00000100:00000200:12.0:1547251615.082948:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c3900 x1622384012445200/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251622 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251615.082959:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:14.0:1547251615.082960:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fbcd [1] + 192 00000800:00000200:12.0:1547251615.082961:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000800:00000200:12.0:1547251615.082962:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:12.0:1547251615.082963:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000100:00000200:14.0:1547251615.082964:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf6f00 x1622384012445184/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251622 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251615.082965:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251615.082966:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251615.082969:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251615.082971:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04620 00000400:00000200:14.0:1547251615.082975:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251615.082977:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000100:00000200:2.0:1547251615.082977:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c3900 x1622384012445200/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251622 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251615.082978:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:14.0:1547251615.082979:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:14.0:1547251615.082981:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:14.0:1547251615.082982:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000100:00000200:2.0:1547251615.082983:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c3900 x1622384012445200/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251622 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251615.082987:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00000400:00000200:12.0:1547251615.082989:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fbdd [1] + 192 00010000:00080000:2.0:1547251615.082990:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c3900 x1622384012445200/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251622 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251615.082993:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf6f00 x1622384012445184/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251622 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251615.082999:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf6f00 x1622384012445184/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251622 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251615.083003:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaaa0 00000100:00000200:12.0:1547251615.083006:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328ddd00 x1622384012445216/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251622 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251615.083008:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf6f00 x1622384012445184/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251622 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:12.0:1547251615.083017:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251615.083019:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251615.083020:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251615.083021:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:1.0:1547251615.083028:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328ddd00 x1622384012445216/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251622 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:1.0:1547251615.083034:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328ddd00 x1622384012445216/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251622 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251615.083038:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00010000:00080000:1.0:1547251615.083041:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328ddd00 x1622384012445216/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251622 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251617.194679:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445232, portal 4 00000100:00000200:8.0:1547251617.194681:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445232, offset 0 00000400:00000200:8.0:1547251617.194684:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251617.194687:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251617.194689:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23038] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251617.194693:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23038] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23038] 00000400:00000200:8.0:1547251617.194695:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251617.194696:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251617.194698:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251617.194699:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9751, 9751] 00000400:00000200:8.0:1547251617.194701:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:8.0:1547251617.194706:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251617.194708:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:8.0:1547251617.194711:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251617.194714:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:8.0:1547251617.194715:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:8.0:1547251617.194718:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000100:00000200:12.0:1547251617.194750:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445248, portal 4 00000100:00000200:12.0:1547251617.194753:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445248, offset 0 00000400:00000200:12.0:1547251617.194756:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:12.0:1547251617.194759:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:12.0:1547251617.194761:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23039] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251617.194764:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23038] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23039] 00000400:00000200:12.0:1547251617.194767:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:5.2:1547251617.194767:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000400:00000200:12.0:1547251617.194768:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251617.194770:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251617.194771:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9751, 9752] 00000400:00000200:12.0:1547251617.194773:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:12.0:1547251617.194778:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251617.194780:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:12.0:1547251617.194783:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251617.194785:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:12.0:1547251617.194786:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:12.0:1547251617.194788:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000800:00000200:11.0:1547251617.194791:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000100:00000200:2.0:1547251617.194792:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445264, portal 4 00000100:00000200:2.0:1547251617.194795:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445264, offset 0 00000400:00000200:2.0:1547251617.194798:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:11.0:1547251617.194800:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000400:00000200:2.0:1547251617.194801:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:2.0:1547251617.194803:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23039] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:11.0:1547251617.194805:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824755d00 x1622384012445232/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251617.194807:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:2.0:1547251617.194807:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23039] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23039] 00000400:00000200:2.0:1547251617.194809:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:2.0:1547251617.194811:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251617.194812:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251617.194813:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000400:00000200:2.0:1547251617.194813:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9752, 9752] 00000400:00000200:11.0:1547251617.194814:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:2.0:1547251617.194815:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251617.194818:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000400:00000200:2.0:1547251617.194820:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251617.194823:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:2.0:1547251617.194823:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:2.0:1547251617.194826:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251617.194827:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:2.0:1547251617.194828:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:2.0:1547251617.194832:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000800:00000200:7.2:1547251617.194839:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000800:00000200:5.2:1547251617.194849:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251617.194855:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251617.194858:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251617.194861:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251617.194863:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000100:00000200:10.0:1547251617.194863:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8caef00 x1622384012445248/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251617.194866:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f5d00 x1622384012445264/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251617.194870:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038aa0 00000400:00000200:11.0:1547251617.194872:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06acc0 00000400:00000200:10.0:1547251617.194872:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251617.194873:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251617.194875:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251617.194876:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251617.194877:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:11.0:1547251617.194878:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251617.194880:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:11.0:1547251617.194881:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:5.2:1547251617.194892:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:10.0:1547251617.194899:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251617.194907:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251617.194912:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000400:00000200:10.0:1547251617.194913:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251617.194914:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:7.2:1547251617.194914:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000400:00000200:10.0:1547251617.194918:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04630 00000400:00000200:11.0:1547251617.194919:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251617.194922:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04650 00000400:00000200:10.0:1547251617.194922:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6e09 [1] + 192 00000400:00000200:11.0:1547251617.194926:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6e39 [1] + 192 00000100:00000200:10.0:1547251617.194931:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824755d00 x1622384012445232/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251617.194949:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251617.194951:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f5d00 x1622384012445264/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251617.194951:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (23)++ 00000800:00000200:10.0:1547251617.194953:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (24)-- 00000800:00000200:10.0:1547251617.194954:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (23)-- 00000800:00000200:10.0:1547251617.194956:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251617.194958:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251617.194960:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251617.194961:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251617.194963:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (21)++ 00000100:00000200:8.0:1547251617.194963:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824755d00 x1622384012445232/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251617.194964:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251617.194964:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251617.194965:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:11.0:1547251617.194967:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:10.0:1547251617.194968:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04640 00000100:00000200:8.0:1547251617.194969:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824755d00 x1622384012445232/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251617.194973:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00010000:00080000:8.0:1547251617.194977:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824755d00 x1622384012445232/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:2.0:1547251617.194977:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f5d00 x1622384012445264/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251617.194982:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f5d00 x1622384012445264/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251617.194987:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00000400:00000200:10.0:1547251617.194989:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6e29 [1] + 192 00010000:00080000:2.0:1547251617.194990:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f5d00 x1622384012445264/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251617.194993:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8caef00 x1622384012445248/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251617.195003:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251617.195004:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251617.195005:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251617.195006:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:12.0:1547251617.195012:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8caef00 x1622384012445248/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251617.195017:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8caef00 x1622384012445248/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251617.195021:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b0383b8 00010000:00080000:12.0:1547251617.195024:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8caef00 x1622384012445248/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:13.0:1547251617.658688:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445280, portal 4 00000100:00000200:13.0:1547251617.658691:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445280, offset 0 00000400:00000200:13.0:1547251617.658695:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:13.0:1547251617.658698:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:13.0:1547251617.658700:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23040] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:9.0:1547251617.658701:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445296, portal 4 00000400:00000200:13.0:1547251617.658704:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23039] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23040] 00000100:00000200:9.0:1547251617.658704:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445296, offset 0 00000400:00000200:13.0:1547251617.658706:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251617.658707:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:13.0:1547251617.658708:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:1.0:1547251617.658708:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445312, portal 4 00000400:00000200:13.0:1547251617.658709:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251617.658710:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9744, 9745] 00000400:00000200:9.0:1547251617.658710:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000100:00000200:1.0:1547251617.658711:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445312, offset 0 00000400:00000200:13.0:1547251617.658712:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:9.0:1547251617.658712:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23040] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251617.658714:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251617.658716:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23040] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23040] 00000400:00000200:13.0:1547251617.658717:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:1.0:1547251617.658717:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251617.658718:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251617.658720:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251617.658720:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23041] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:13.0:1547251617.658721:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:9.0:1547251617.658722:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251617.658723:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9745, 9745] 00000800:00000200:13.0:1547251617.658724:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:1.0:1547251617.658724:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23040] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23041] 00000800:00000200:13.0:1547251617.658725:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000400:00000200:9.0:1547251617.658725:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:13.0:1547251617.658726:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:1.0:1547251617.658726:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:1.0:1547251617.658727:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:13.0:1547251617.658729:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:1.0:1547251617.658729:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251617.658730:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:1.0:1547251617.658731:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9745, 9746] 00000800:00000200:9.0:1547251617.658733:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:1.0:1547251617.658733:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:9.0:1547251617.658736:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251617.658737:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:9.0:1547251617.658738:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000400:00000200:1.0:1547251617.658739:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251617.658741:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:1.0:1547251617.658744:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:0.2:1547251617.658746:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:1.0:1547251617.658748:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251617.658751:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:1.0:1547251617.658753:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:1.0:1547251617.658756:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:12.2:1547251617.658757:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:12.2:1547251617.658763:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251617.658807:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251617.658811:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251617.658811:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251617.658813:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000100:00000200:5.0:1547251617.658815:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af1b00 x1622384012445280/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251617.658817:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf6c00 x1622384012445296/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251617.658823:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b2a8 00000400:00000200:4.0:1547251617.658823:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eabb0 00000400:00000200:5.0:1547251617.658824:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251617.658824:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251617.658828:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251617.658828:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251617.658831:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251617.658832:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251617.658833:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251617.658834:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:4.0:1547251617.658834:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251617.658835:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:5.0:1547251617.658837:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000100:00000200:4.0:1547251617.658837:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b7b00 x1622384012445312/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251617.658843:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2330 00000400:00000200:4.0:1547251617.658843:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251617.658846:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251617.658848:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:12.2:1547251617.658903:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:0.2:1547251617.658912:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:5.0:1547251617.658919:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251617.658923:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000800:00000200:12.2:1547251617.658925:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:4.0:1547251617.658927:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:5.0:1547251617.658928:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251617.658930:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251617.658933:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04660 00000400:00000200:4.0:1547251617.658934:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251617.658937:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3c35 [1] + 192 00000400:00000200:4.0:1547251617.658938:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04680 00000400:00000200:4.0:1547251617.658950:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3c55 [1] + 192 00000100:00000200:4.0:1547251617.658955:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b7b00 x1622384012445312/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251617.658968:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251617.658970:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af1b00 x1622384012445280/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251617.658971:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251617.658973:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251617.658976:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251617.658978:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:5.0:1547251617.658980:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251617.658980:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251617.658982:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251617.658982:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000100:00000200:1.0:1547251617.658982:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b7b00 x1622384012445312/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251617.658984:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251617.658985:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:5.0:1547251617.658987:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000400:00000200:4.0:1547251617.658987:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:5.0:1547251617.658988:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:1.0:1547251617.658988:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b7b00 x1622384012445312/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251617.658990:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04670 00000400:00000200:1.0:1547251617.658992:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00000100:00000200:13.0:1547251617.658996:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af1b00 x1622384012445280/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:1.0:1547251617.658996:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b7b00 x1622384012445312/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:13.0:1547251617.659002:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af1b00 x1622384012445280/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251617.659006:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b3b8 00000400:00000200:4.0:1547251617.659009:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3c45 [1] + 192 00010000:00080000:13.0:1547251617.659010:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af1b00 x1622384012445280/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251617.659012:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf6c00 x1622384012445296/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251617.659020:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251617.659021:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251617.659022:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251617.659024:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:9.0:1547251617.659070:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf6c00 x1622384012445296/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251617.659076:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf6c00 x1622384012445296/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251617.659081:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaaa0 00010000:00080000:9.0:1547251617.659085:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf6c00 x1622384012445296/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251624 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251618.058686:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445328, portal 4 00000100:00000200:9.0:1547251618.058689:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445328, offset 0 00000400:00000200:9.0:1547251618.058692:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251618.058694:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251618.058697:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04690 00000400:00000200:9.0:1547251618.058700:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd46e9 [1] + 0 00000400:00000200:9.0:1547251618.058703:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ba9d367990 00000100:00000200:9.0:1547251618.058704:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251618.058713:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba9d367990 00000400:00000200:9.0:1547251618.058714:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251618.058718:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf5a00 x1622384012445328/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251625 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251618.058724:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea4c8 00000400:00000200:9.0:1547251618.058725:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251618.058727:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251618.058736:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445328 00000100:00000200:6.0:1547251618.058741:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445344, portal 4 00000100:00000200:6.0:1547251618.058744:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445344, offset 0 00010000:00000200:8.0:1547251618.058745:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3db050 x1622384012445328/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:679/0 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:6.0:1547251618.058748:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251618.058751:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:8.0:1547251618.058752:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012445328, offset 192 00000400:00000200:8.0:1547251618.058754:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251618.058754:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c046a0 00000400:00000200:8.0:1547251618.058757:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251618.058757:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe24e5 [1] + 0 00000400:00000200:6.0:1547251618.058759:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990330 00000400:00000200:8.0:1547251618.058760:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04690 00000100:00000200:6.0:1547251618.058760:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251618.058762:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaef61 [1] + 192 00000100:00000200:8.0:1547251618.058765:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf5a00 x1622384012445328/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251625 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251618.058770:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990330 00000400:00000200:6.0:1547251618.058771:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251618.058773:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251618.058773:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b82444fb00 x1622384012445344/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251625 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251618.058775:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6886e8 00000400:00000200:8.0:1547251618.058776:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251618.058778:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990b28 00000400:00000200:8.0:1547251618.058779:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251618.058779:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251618.058780:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251618.058783:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf5a00 x1622384012445328/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251625 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251618.058785:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:9.0:1547251618.058789:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf5a00 x1622384012445328/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251625 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251618.058791:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445344 00000400:00000200:9.0:1547251618.058794:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaaa0 00010000:00080000:9.0:1547251618.058798:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf5a00 x1622384012445328/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251625 ref 1 fl Rpc:R/0/0 rc 0/0 00010000:00000200:4.0:1547251618.058800:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39bc50 x1622384012445344/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:679/0 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251618.058807:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012445344, offset 192 00000400:00000200:4.0:1547251618.058809:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251618.058812:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251618.058815:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c046a0 00000400:00000200:4.0:1547251618.058818:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaef71 [1] + 192 00000100:00000200:4.0:1547251618.058821:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b82444fb00 x1622384012445344/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251625 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251618.058829:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251618.058831:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bdd0 00000400:00000200:4.0:1547251618.058832:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251618.058833:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251618.058838:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b82444fb00 x1622384012445344/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251625 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251618.058838:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251618.058843:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b82444fb00 x1622384012445344/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251625 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251618.058847:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459905d8 00010000:00080000:6.0:1547251618.058850:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b82444fb00 x1622384012445344/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251625 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251618.074738:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445360, portal 4 00000100:00000200:5.0:1547251618.074743:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445360, offset 0 00000400:00000200:5.0:1547251618.074747:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251618.074750:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251618.074752:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c046b0 00000400:00000200:5.0:1547251618.074755:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2515 [1] + 0 00000400:00000200:5.0:1547251618.074757:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990aa0 00000100:00000200:5.0:1547251618.074759:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:5.0:1547251618.074767:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990aa0 00000400:00000200:5.0:1547251618.074769:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251618.074773:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37936900 x1622384012445360/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251625 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251618.074779:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad198 00000400:00000200:5.0:1547251618.074780:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251618.074782:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251618.074792:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445360 00010000:00000200:4.0:1547251618.074802:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39b850 x1622384012445360/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:679/0 lens 224/368 e 0 to 0 dl 1547251624 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251618.074809:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012445360, offset 192 00000400:00000200:4.0:1547251618.074811:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251618.074815:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251618.074817:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c046b0 00000400:00000200:4.0:1547251618.074820:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaef81 [1] + 192 00000100:00000200:4.0:1547251618.074822:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37936900 x1622384012445360/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251625 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251618.074832:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251618.074834:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b550 00000400:00000200:4.0:1547251618.074835:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251618.074838:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251618.074843:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37936900 x1622384012445360/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251625 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251618.074844:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251618.074849:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37936900 x1622384012445360/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251625 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251618.074854:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad2a8 00010000:00080000:5.0:1547251618.074858:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37936900 x1622384012445360/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251625 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251620.090676:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445376, portal 4 00000100:00000200:8.0:1547251620.090679:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445376, offset 0 00000400:00000200:8.0:1547251620.090681:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:8.0:1547251620.090684:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:8.0:1547251620.090686:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23042] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251620.090690:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23040] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23042] 00000400:00000200:8.0:1547251620.090692:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:8.0:1547251620.090694:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251620.090696:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251620.090697:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9746, 9747] 00000400:00000200:8.0:1547251620.090699:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:8.0:1547251620.090703:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251620.090706:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:8.0:1547251620.090709:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251620.090710:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:8.0:1547251620.090711:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:8.0:1547251620.090714:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:3.0:1547251620.090720:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445392, portal 4 00000100:00000200:3.0:1547251620.090723:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445392, offset 0 00000400:00000200:3.0:1547251620.090728:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:9.2:1547251620.090731:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:3.0:1547251620.090731:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:3.0:1547251620.090733:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23042] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251620.090736:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23041] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23042] 00000400:00000200:3.0:1547251620.090739:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251620.090740:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251620.090741:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251620.090743:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9747, 9747] 00000400:00000200:3.0:1547251620.090744:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:3.0:1547251620.090749:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251620.090752:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:3.0:1547251620.090755:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251620.090756:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:3.0:1547251620.090757:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:3.0:1547251620.090760:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:14.0:1547251620.090784:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000100:00000200:5.0:1547251620.090787:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445408, portal 4 00000100:00000200:5.0:1547251620.090790:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445408, offset 0 00000400:00000200:5.0:1547251620.090793:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:14.0:1547251620.090794:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000400:00000200:5.0:1547251620.090796:0:12674:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000100:00000200:14.0:1547251620.090797:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824750000 x1622384012445376/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251627 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251620.090799:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23043] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:5.0:1547251620.090802:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23041] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23043] 00000400:00000200:14.0:1547251620.090803:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000400:00000200:14.0:1547251620.090804:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:7.2:1547251620.090804:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000400:00000200:5.0:1547251620.090804:0:12674:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:14.0:1547251620.090806:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:5.0:1547251620.090806:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:5.0:1547251620.090807:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:5.0:1547251620.090808:0:12674:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9747, 9748] 00000800:00000200:14.0:1547251620.090809:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:5.0:1547251620.090810:0:12674:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000800:00000200:14.0:1547251620.090811:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:14.0:1547251620.090812:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:14.0:1547251620.090814:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000100:00000200:14.0:1547251620.090816:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328ddd00 x1622384012445392/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251627 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251620.090816:0:12674:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:5.0:1547251620.090819:0:12674:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:14.0:1547251620.090820:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555bb0 00000400:00000200:14.0:1547251620.090821:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251620.090823:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:5.0:1547251620.090823:0:12674:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251620.090824:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:5.0:1547251620.090824:0:12674:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:5.0:1547251620.090825:0:12674:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:14.0:1547251620.090826:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:5.0:1547251620.090828:0:12674:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:9.2:1547251620.090840:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:14.0:1547251620.090851:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:14.0:1547251620.090858:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000100:00000200:14.0:1547251620.090860:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b82444b300 x1622384012445408/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251627 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251620.090864:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad198 00000400:00000200:14.0:1547251620.090865:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:12.0:1547251620.090865:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:14.0:1547251620.090867:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251620.090869:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:9.2:1547251620.090899:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:7.2:1547251620.090904:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:12.0:1547251620.090912:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251620.090916:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.93@o2ib10 00000800:00000200:14.0:1547251620.090917:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:14.0:1547251620.090920:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251620.090920:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251620.090923:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251620.090925:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c046c0 00000400:00000200:14.0:1547251620.090927:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c046d0 00000400:00000200:12.0:1547251620.090928:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fc0d [1] + 192 00000400:00000200:14.0:1547251620.090931:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fc1d [1] + 192 00000100:00000200:12.0:1547251620.090936:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824750000 x1622384012445376/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251627 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251620.090954:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:14.0:1547251620.090955:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328ddd00 x1622384012445392/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251627 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251620.090956:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251620.090958:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251620.090959:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251620.090960:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251620.090962:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251620.090964:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:12.0:1547251620.090965:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:14.0:1547251620.090966:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000100:00000200:8.0:1547251620.090966:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824750000 x1622384012445376/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251627 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251620.090967:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c046e0 00000800:00000200:14.0:1547251620.090968:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:14.0:1547251620.090969:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:14.0:1547251620.090970:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:14.0:1547251620.090971:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:8.0:1547251620.090972:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824750000 x1622384012445376/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251627 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251620.090976:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000400:00000200:12.0:1547251620.090978:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fc2d [1] + 192 00010000:00080000:8.0:1547251620.090979:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824750000 x1622384012445376/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251627 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251620.090982:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328ddd00 x1622384012445392/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251627 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251620.090987:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328ddd00 x1622384012445392/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251627 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251620.090996:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00000100:00000200:12.0:1547251620.090999:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b82444b300 x1622384012445408/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251627 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:3.0:1547251620.091000:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328ddd00 x1622384012445392/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251627 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:12.0:1547251620.091006:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251620.091008:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251620.091009:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251620.091009:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000100:00000200:5.0:1547251620.091016:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b82444b300 x1622384012445408/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251627 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251620.091022:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b82444b300 x1622384012445408/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251627 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251620.091026:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad2a8 00010000:00080000:5.0:1547251620.091030:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b82444b300 x1622384012445408/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251627 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251622.202687:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445424, portal 4 00000100:00000200:13.0:1547251622.202689:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445440, portal 4 00000100:00000200:9.0:1547251622.202690:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445424, offset 0 00000100:00000200:13.0:1547251622.202692:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445440, offset 0 00000400:00000200:9.0:1547251622.202693:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:13.0:1547251622.202695:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251622.202696:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:13.0:1547251622.202698:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251622.202698:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23043] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:13.0:1547251622.202700:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23043] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251622.202702:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23042] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23043] 00000400:00000200:13.0:1547251622.202703:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23042] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23043] 00000400:00000200:9.0:1547251622.202704:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:13.0:1547251622.202706:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251622.202706:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251622.202707:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251622.202707:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251622.202708:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9752, 9753] 00000400:00000200:13.0:1547251622.202710:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251622.202710:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:13.0:1547251622.202712:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9753, 9753] 00000400:00000200:13.0:1547251622.202714:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:9.0:1547251622.202715:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251622.202718:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:13.0:1547251622.202721:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251622.202723:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:9.0:1547251622.202725:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251622.202726:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251622.202727:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:13.0:1547251622.202728:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:9.0:1547251622.202728:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:13.0:1547251622.202729:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:9.0:1547251622.202731:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000800:00000200:13.0:1547251622.202732:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000100:00000200:0.0:1547251622.202774:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445456, portal 4 00000100:00000200:0.0:1547251622.202778:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445456, offset 0 00000400:00000200:0.0:1547251622.202782:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:0.0:1547251622.202786:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:0.0:1547251622.202789:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23044] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251622.202792:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23043] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23044] 00000800:00000200:7.2:1547251622.202793:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000400:00000200:0.0:1547251622.202794:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251622.202795:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251622.202797:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251622.202798:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9753, 9754] 00000400:00000200:0.0:1547251622.202800:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000800:00000200:5.2:1547251622.202803:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000400:00000200:0.0:1547251622.202804:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251622.202807:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:0.0:1547251622.202810:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251622.202813:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (22)++ 00000800:00000200:0.0:1547251622.202814:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (23)++ 00000800:00000200:0.0:1547251622.202817:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (24)-- 00000800:00000200:11.0:1547251622.202819:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:10.0:1547251622.202821:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251622.202822:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000800:00000200:10.0:1547251622.202823:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000100:00000200:11.0:1547251622.202826:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf5400 x1622384012445424/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:10.0:1547251622.202828:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af2700 x1622384012445440/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251622.202834:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea4c8 00000400:00000200:10.0:1547251622.202834:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792bd48 00000400:00000200:11.0:1547251622.202835:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251622.202836:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251622.202839:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251622.202839:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251622.202842:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251622.202843:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251622.202844:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251622.202845:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251622.202845:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:10.0:1547251622.202847:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000100:00000200:11.0:1547251622.202848:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c3900 x1622384012445456/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251622.202854:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665c38 00000400:00000200:11.0:1547251622.202855:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251622.202859:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251622.202860:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:5.2:1547251622.202912:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251622.202917:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:7.2:1547251622.202923:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:10.0:1547251622.202925:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251622.202929:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251622.202930:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251622.202933:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251622.202934:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251622.202937:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251622.202939:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c046f0 00000400:00000200:11.0:1547251622.202950:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04700 00000400:00000200:10.0:1547251622.202952:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6e99 [1] + 192 00000400:00000200:11.0:1547251622.202956:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6ea9 [1] + 192 00000100:00000200:10.0:1547251622.202961:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf5400 x1622384012445424/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251622.202973:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251622.202975:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af2700 x1622384012445440/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251622.202975:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:10.0:1547251622.202977:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251622.202979:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251622.202981:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251622.202983:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251622.202986:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251622.202987:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000400:00000200:10.0:1547251622.202987:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000100:00000200:9.0:1547251622.202988:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf5400 x1622384012445424/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251622.202989:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:11.0:1547251622.202990:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:10.0:1547251622.202991:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04710 00000800:00000200:11.0:1547251622.202992:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:9.0:1547251622.202993:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf5400 x1622384012445424/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251622.202994:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:11.0:1547251622.202995:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:9.0:1547251622.202997:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaaa0 00000400:00000200:10.0:1547251622.203001:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6ec9 [1] + 192 00010000:00080000:9.0:1547251622.203001:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf5400 x1622384012445424/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:13.0:1547251622.203007:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af2700 x1622384012445440/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:13.0:1547251622.203012:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af2700 x1622384012445440/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251622.203016:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b3b8 00000100:00000200:10.0:1547251622.203019:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c3900 x1622384012445456/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:13.0:1547251622.203020:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af2700 x1622384012445440/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251622.203027:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251622.203028:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251622.203029:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251622.203030:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:0.0:1547251622.203037:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c3900 x1622384012445456/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251622.203043:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c3900 x1622384012445456/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251622.203047:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665d48 00010000:00080000:0.0:1547251622.203051:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c3900 x1622384012445456/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:12.0:1547251622.666678:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445488, portal 4 00000100:00000200:12.0:1547251622.666681:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445488, offset 0 00000400:00000200:12.0:1547251622.666684:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000100:00000200:8.0:1547251622.666684:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445472, portal 4 00000400:00000200:12.0:1547251622.666687:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000100:00000200:8.0:1547251622.666687:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445472, offset 0 00000400:00000200:12.0:1547251622.666689:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23044] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251622.666690:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:12.0:1547251622.666693:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23044] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23044] 00000400:00000200:8.0:1547251622.666693:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:12.0:1547251622.666695:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251622.666695:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23044] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251622.666697:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251622.666698:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251622.666699:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9746, 9746] 00000400:00000200:8.0:1547251622.666699:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23044] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23045] 00000400:00000200:8.0:1547251622.666701:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:12.0:1547251622.666702:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:8.0:1547251622.666703:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251622.666704:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251622.666706:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9746, 9747] 00000100:00000200:0.0:1547251622.666706:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445504, portal 4 00000400:00000200:12.0:1547251622.666707:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:8.0:1547251622.666708:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000100:00000200:0.0:1547251622.666709:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445504, offset 0 00000800:00000200:12.0:1547251622.666710:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:0.0:1547251622.666712:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:12.0:1547251622.666713:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:8.0:1547251622.666713:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251622.666714:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:12.0:1547251622.666715:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:8.0:1547251622.666715:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:0.0:1547251622.666715:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:0.0:1547251622.666717:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23045] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:12.0:1547251622.666718:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:8.0:1547251622.666718:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:0.0:1547251622.666720:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23045] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23045] 00000800:00000200:8.0:1547251622.666721:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:8.0:1547251622.666722:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:0.0:1547251622.666723:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:0.0:1547251622.666724:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251622.666725:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:0.0:1547251622.666726:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:12.2:1547251622.666727:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000400:00000200:0.0:1547251622.666727:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9747, 9747] 00000400:00000200:0.0:1547251622.666729:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:0.2:1547251622.666735:0:12672:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:0.0:1547251622.666752:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251622.666755:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:0.0:1547251622.666758:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251622.666759:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (22)++ 00000800:00000200:0.0:1547251622.666760:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00000800:00000200:0.0:1547251622.666763:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (24)-- 00000800:00000200:4.0:1547251622.666785:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:5.0:1547251622.666789:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251622.666789:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000800:00000200:5.0:1547251622.666792:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:4.0:1547251622.666794:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f1b00 x1622384012445488/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251622.666795:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824755100 x1622384012445472/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251622.666802:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000400:00000200:4.0:1547251622.666802:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038cc0 00000400:00000200:5.0:1547251622.666803:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251622.666803:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251622.666807:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251622.666807:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251622.666810:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251622.666811:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251622.666812:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251622.666813:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:4.0:1547251622.666814:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251622.666815:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:4.0:1547251622.666816:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328de000 x1622384012445504/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251622.666822:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665440 00000400:00000200:4.0:1547251622.666823:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251622.666826:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251622.666828:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:12.2:1547251622.666904:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:0.2:1547251622.666909:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:5.0:1547251622.666919:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251622.666922:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251622.666923:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251622.666925:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251622.666927:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251622.666929:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251622.666932:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04720 00000400:00000200:4.0:1547251622.666934:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04730 00000400:00000200:5.0:1547251622.666936:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3ce5 [1] + 192 00000400:00000200:4.0:1547251622.666950:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3cd5 [1] + 192 00000100:00000200:5.0:1547251622.666961:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824755100 x1622384012445472/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251622.666973:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251622.666975:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f1b00 x1622384012445488/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251622.666977:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251622.666979:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251622.666981:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251622.666983:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:5.0:1547251622.666984:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:5.0:1547251622.666987:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251622.666991:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251622.666992:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251622.666994:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824755100 x1622384012445472/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251622.666994:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)++ 00000400:00000200:5.0:1547251622.666995:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04740 00000800:00000200:4.0:1547251622.666995:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (24)-- 00000800:00000200:4.0:1547251622.666997:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:4.0:1547251622.666998:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000100:00000200:8.0:1547251622.666999:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824755100 x1622384012445472/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251622.667003:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000400:00000200:5.0:1547251622.667006:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3cf5 [1] + 192 00010000:00080000:8.0:1547251622.667007:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824755100 x1622384012445472/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:12.0:1547251622.667012:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f1b00 x1622384012445488/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251622.667017:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f1b00 x1622384012445488/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251622.667021:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b0383b8 00000100:00000200:5.0:1547251622.667024:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328de000 x1622384012445504/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:12.0:1547251622.667025:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f1b00 x1622384012445488/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251622.667034:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251622.667035:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251622.667036:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251622.667037:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000100:00000200:0.0:1547251622.667044:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328de000 x1622384012445504/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251622.667049:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328de000 x1622384012445504/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251622.667058:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665d48 00010000:00080000:0.0:1547251622.667061:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328de000 x1622384012445504/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251629 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251623.066677:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445520, portal 4 00000100:00000200:8.0:1547251623.066680:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445520, offset 0 00000400:00000200:8.0:1547251623.066682:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251623.066685:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251623.066688:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04750 00000400:00000200:8.0:1547251623.066690:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4759 [1] + 0 00000400:00000200:8.0:1547251623.066692:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ba9d367660 00000100:00000200:8.0:1547251623.066694:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251623.066703:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba9d367660 00000400:00000200:8.0:1547251623.066703:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251623.066706:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824752d00 x1622384012445520/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251630 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251623.066711:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000400:00000200:8.0:1547251623.066711:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251623.066712:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:11.0:1547251623.066736:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445520 00000100:00000200:7.0:1547251623.066736:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445536, portal 4 00000100:00000200:7.0:1547251623.066739:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445536, offset 0 00000400:00000200:7.0:1547251623.066741:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251623.066744:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00010000:00000200:11.0:1547251623.066747:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3dac50 x1622384012445520/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:684/0 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:7.0:1547251623.066747:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04760 00000400:00000200:7.0:1547251623.066749:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2585 [1] + 0 00000400:00000200:7.0:1547251623.066751:0:12676:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990a18 00000100:00000200:7.0:1547251623.066753:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:11.0:1547251623.066756:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012445520, offset 192 00000400:00000200:11.0:1547251623.066759:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:11.0:1547251623.066762:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251623.066763:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990a18 00000400:00000200:7.0:1547251623.066764:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:11.0:1547251623.066766:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04750 00000400:00000200:11.0:1547251623.066768:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaefc1 [1] + 192 00000100:00000200:7.0:1547251623.066768:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df5100 x1622384012445536/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251630 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251623.066771:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824752d00 x1622384012445520/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251630 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251623.066775:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea2a8 00000400:00000200:7.0:1547251623.066775:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251623.066777:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:11.0:1547251623.066781:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:11.0:1547251623.066783:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bee0 00000400:00000200:11.0:1547251623.066784:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251623.066785:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251623.066787:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445536 00000100:00000200:8.0:1547251623.066789:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824752d00 x1622384012445520/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251630 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251623.066790:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:8.0:1547251623.066794:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824752d00 x1622384012445520/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251630 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251623.066797:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00010000:00000200:6.0:1547251623.066797:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39b450 x1622384012445536/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:684/0 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Interpret:/0/0 rc 0/0 00010000:00080000:8.0:1547251623.066800:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824752d00 x1622384012445520/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251630 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251623.066804:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012445536, offset 192 00000400:00000200:6.0:1547251623.066806:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251623.066809:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251623.066812:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04760 00000400:00000200:6.0:1547251623.066815:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaefd1 [1] + 192 00000100:00000200:6.0:1547251623.066818:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df5100 x1622384012445536/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251630 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251623.066827:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251623.066829:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459905d8 00000400:00000200:6.0:1547251623.066830:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251623.066832:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251623.066837:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df5100 x1622384012445536/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251630 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251623.066839:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251623.066843:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df5100 x1622384012445536/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251630 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251623.066848:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea908 00010000:00080000:7.0:1547251623.066852:0:12676:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df5100 x1622384012445536/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251630 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251623.082747:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445552, portal 4 00000100:00000200:4.0:1547251623.082750:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445552, offset 0 00000400:00000200:4.0:1547251623.082752:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251623.082755:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251623.082758:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04770 00000400:00000200:4.0:1547251623.082761:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe25b5 [1] + 0 00000400:00000200:4.0:1547251623.082763:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17ea550 00000100:00000200:4.0:1547251623.082765:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:4.0:1547251623.082773:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea550 00000400:00000200:4.0:1547251623.082774:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251623.082776:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37933900 x1622384012445552/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251630 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251623.082781:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b330 00000400:00000200:4.0:1547251623.082781:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251623.082783:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251623.082793:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445552 00010000:00000200:6.0:1547251623.082802:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39b050 x1622384012445552/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:684/0 lens 224/368 e 0 to 0 dl 1547251629 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251623.082809:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012445552, offset 192 00000400:00000200:6.0:1547251623.082812:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251623.082815:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251623.082817:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04770 00000400:00000200:6.0:1547251623.082820:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaefe1 [1] + 192 00000100:00000200:6.0:1547251623.082823:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37933900 x1622384012445552/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251630 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251623.082832:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251623.082834:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990b28 00000400:00000200:6.0:1547251623.082835:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251623.082836:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251623.082841:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37933900 x1622384012445552/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251630 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251623.082842:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:4.0:1547251623.082845:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37933900 x1622384012445552/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251630 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251623.082849:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b088 00010000:00080000:4.0:1547251623.082852:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37933900 x1622384012445552/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251630 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251625.098770:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445568, portal 4 00000100:00000200:11.0:1547251625.098773:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445568, offset 0 00000400:00000200:11.0:1547251625.098777:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:11.0:1547251625.098780:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:11.0:1547251625.098782:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23046] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:11.0:1547251625.098786:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23045] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23046] 00000400:00000200:11.0:1547251625.098788:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:11.0:1547251625.098790:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251625.098792:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251625.098793:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9748, 9748] 00000400:00000200:11.0:1547251625.098795:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:11.0:1547251625.098800:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000100:00000200:7.0:1547251625.098800:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445584, portal 4 00000800:00000200:11.0:1547251625.098802:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000100:00000200:2.0:1547251625.098802:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445600, portal 4 00000100:00000200:7.0:1547251625.098804:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445584, offset 0 00000800:00000200:11.0:1547251625.098805:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000100:00000200:2.0:1547251625.098805:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445600, offset 0 00000800:00000200:11.0:1547251625.098807:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:11.0:1547251625.098808:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:7.0:1547251625.098808:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:2.0:1547251625.098808:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:11.0:1547251625.098811:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:7.0:1547251625.098811:0:12675:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:2.0:1547251625.098811:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:7.0:1547251625.098814:0:12675:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23046] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251625.098814:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23046] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251625.098817:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23046] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23046] 00000400:00000200:7.0:1547251625.098818:0:12675:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23046] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23046] 00000400:00000200:7.0:1547251625.098820:0:12675:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:2.0:1547251625.098820:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:7.0:1547251625.098822:0:12675:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251625.098822:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251625.098823:0:12675:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251625.098823:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251625.098825:0:12675:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9748, 9749] 00000400:00000200:2.0:1547251625.098825:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9748, 9749] 00000800:00000200:9.2:1547251625.098826:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:7.0:1547251625.098827:0:12675:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:2.0:1547251625.098827:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:7.0:1547251625.098831:0:12675:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:7.0:1547251625.098834:0:12675:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:7.0:1547251625.098837:0:12675:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000400:00000200:2.0:1547251625.098837:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:7.0:1547251625.098839:0:12675:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:7.0:1547251625.098840:0:12675:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:2.0:1547251625.098840:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:7.0:1547251625.098843:0:12675:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:2.0:1547251625.098843:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251625.098844:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:2.0:1547251625.098846:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:2.0:1547251625.098848:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:7.2:1547251625.098851:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:14.0:1547251625.098879:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251625.098881:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)++ 00000800:00000200:14.0:1547251625.098882:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:14.0:1547251625.098885:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824750900 x1622384012445568/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251632 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251625.098885:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (24)-- 00000100:00000200:12.0:1547251625.098889:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c430900 x1622384012445584/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251632 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251625.098891:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0be58 00000400:00000200:14.0:1547251625.098892:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251625.098894:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea2a8 00000400:00000200:14.0:1547251625.098895:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:12.0:1547251625.098896:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:14.0:1547251625.098897:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:12.0:1547251625.098898:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251625.098899:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:14.0:1547251625.098900:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)++ 00000800:00000200:14.0:1547251625.098901:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (24)-- 00000800:00000200:12.0:1547251625.098901:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (24)-- 00000800:00000200:12.0:1547251625.098903:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:14.0:1547251625.098904:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f5a00 x1622384012445600/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251632 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251625.098913:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a990 00000400:00000200:14.0:1547251625.098914:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251625.098916:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251625.098917:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:9.2:1547251625.098984:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.2:1547251625.098990:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:12.0:1547251625.098997:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251625.099000:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.93@o2ib10 00000800:00000200:14.0:1547251625.099003:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:12.0:1547251625.099003:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:14.0:1547251625.099005:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251625.099007:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04780 00000400:00000200:14.0:1547251625.099009:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251625.099010:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fc5d [1] + 192 00000400:00000200:14.0:1547251625.099012:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04790 00000100:00000200:12.0:1547251625.099014:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824750900 x1622384012445568/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251632 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251625.099025:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:14.0:1547251625.099026:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fc6d [1] + 192 00000800:00000200:12.0:1547251625.099028:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251625.099029:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:14.0:1547251625.099030:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c430900 x1622384012445584/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251632 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251625.099030:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251625.099032:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:12.0:1547251625.099033:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251625.099035:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251625.099038:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251625.099041:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:12.0:1547251625.099041:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c047a0 00000800:00000200:14.0:1547251625.099043:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)++ 00000100:00000200:11.0:1547251625.099043:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824750900 x1622384012445568/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251632 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251625.099044:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (24)-- 00000800:00000200:14.0:1547251625.099045:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:14.0:1547251625.099046:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:11.0:1547251625.099048:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824750900 x1622384012445568/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251632 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251625.099052:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b220 00000400:00000200:12.0:1547251625.099055:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fc7d [1] + 192 00010000:00080000:11.0:1547251625.099056:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824750900 x1622384012445568/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251632 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251625.099059:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c430900 x1622384012445584/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251632 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251625.099064:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c430900 x1622384012445584/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251632 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251625.099069:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea908 00000100:00000200:12.0:1547251625.099071:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f5a00 x1622384012445600/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251632 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:7.0:1547251625.099073:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ae5c430900 x1622384012445584/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251632 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:12.0:1547251625.099079:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251625.099081:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251625.099082:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251625.099083:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000100:00000200:2.0:1547251625.099089:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f5a00 x1622384012445600/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251632 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251625.099094:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f5a00 x1622384012445600/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251632 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251625.099099:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00010000:00080000:2.0:1547251625.099102:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f5a00 x1622384012445600/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251632 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:1.0:1547251627.210697:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445632, portal 4 00000100:00000200:1.0:1547251627.210700:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445632, offset 0 00000400:00000200:1.0:1547251627.210703:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:1.0:1547251627.210706:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:1.0:1547251627.210708:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23048] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:13.0:1547251627.210710:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445648, portal 4 00000100:00000200:11.0:1547251627.210711:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445616, portal 4 00000400:00000200:1.0:1547251627.210711:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23046] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23048] 00000400:00000200:1.0:1547251627.210713:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000100:00000200:13.0:1547251627.210714:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445648, offset 0 00000100:00000200:11.0:1547251627.210715:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445616, offset 0 00000400:00000200:1.0:1547251627.210715:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251627.210716:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251627.210717:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:1.0:1547251627.210718:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9754, 9754] 00000400:00000200:11.0:1547251627.210719:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:13.0:1547251627.210720:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:1.0:1547251627.210720:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:13.0:1547251627.210722:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23048] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:11.0:1547251627.210722:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:11.0:1547251627.210724:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23048] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:13.0:1547251627.210725:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23047] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23048] 00000400:00000200:13.0:1547251627.210728:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:11.0:1547251627.210728:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23047] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23048] 00000400:00000200:13.0:1547251627.210729:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251627.210730:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:13.0:1547251627.210731:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251627.210731:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251627.210732:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9754, 9755] 00000400:00000200:11.0:1547251627.210733:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251627.210734:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251627.210734:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9754, 9755] 00000400:00000200:11.0:1547251627.210736:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251627.210741:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251627.210743:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:1.0:1547251627.210746:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:1.0:1547251627.210749:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:13.0:1547251627.210752:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:1.0:1547251627.210752:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251627.210753:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:13.0:1547251627.210755:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:1.0:1547251627.210755:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251627.210756:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251627.210757:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:13.0:1547251627.210758:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251627.210758:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:13.0:1547251627.210760:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:11.0:1547251627.210760:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:13.0:1547251627.210762:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251627.210762:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000800:00000200:13.0:1547251627.210764:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000800:00000200:7.2:1547251627.210808:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:7.2:1547251627.210817:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000800:00000200:5.2:1547251627.210825:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251627.210827:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:11.0:1547251627.210830:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251627.210830:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251627.210832:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000100:00000200:11.0:1547251627.210833:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b7b00 x1622384012445632/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:10.0:1547251627.210837:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af6000 x1622384012445648/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251627.210840:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e26e8 00000400:00000200:11.0:1547251627.210842:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251627.210843:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792bd48 00000400:00000200:10.0:1547251627.210844:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251627.210845:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251627.210847:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251627.210848:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251627.210849:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:11.0:1547251627.210850:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251627.210851:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000800:00000200:10.0:1547251627.210851:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:10.0:1547251627.210853:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:11.0:1547251627.210854:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824753f00 x1622384012445616/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251627.210854:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:11.0:1547251627.210859:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bd48 00000400:00000200:11.0:1547251627.210860:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251627.210863:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251627.210865:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:5.2:1547251627.210927:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251627.210935:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:5.2:1547251627.210940:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:10.0:1547251627.210957:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251627.210958:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251627.210960:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251627.210961:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251627.210965:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251627.210965:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251627.210969:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c047b0 00000400:00000200:10.0:1547251627.210969:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c047c0 00000400:00000200:11.0:1547251627.210973:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6f49 [1] + 192 00000400:00000200:10.0:1547251627.210977:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6f29 [1] + 192 00000100:00000200:11.0:1547251627.210980:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824753f00 x1622384012445616/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251627.210993:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251627.210995:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000100:00000200:10.0:1547251627.210995:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b7b00 x1622384012445632/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251627.210997:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:11.0:1547251627.210998:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251627.211000:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251627.211002:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251627.211006:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251627.211006:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251627.211008:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251627.211009:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000400:00000200:11.0:1547251627.211011:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c047d0 00000800:00000200:10.0:1547251627.211011:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251627.211012:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000400:00000200:11.0:1547251627.211014:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd6f39 [1] + 192 00000800:00000200:10.0:1547251627.211014:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:10.0:1547251627.211015:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000100:00000200:1.0:1547251627.211020:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b7b00 x1622384012445632/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:1.0:1547251627.211025:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b7b00 x1622384012445632/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251627.211034:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00000100:00000200:11.0:1547251627.211037:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af6000 x1622384012445648/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:1.0:1547251627.211038:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b7b00 x1622384012445632/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:11.0:1547251627.211047:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251627.211048:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251627.211050:0:12663:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (22)++ 00000100:00000200:8.0:1547251627.211050:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824753f00 x1622384012445616/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251627.211051:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251627.211052:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000100:00000200:8.0:1547251627.211056:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824753f00 x1622384012445616/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251627.211061:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b220 00000800:00000200:5.2:1547251627.211063:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00010000:00080000:8.0:1547251627.211065:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824753f00 x1622384012445616/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:10.0:1547251627.211076:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000800:00000200:10.0:1547251627.211082:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251627.211083:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251627.211085:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:13.0:1547251627.211106:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af6000 x1622384012445648/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:13.0:1547251627.211112:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af6000 x1622384012445648/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251627.211116:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b3b8 00010000:00080000:13.0:1547251627.211120:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af6000 x1622384012445648/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251627.674715:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445664, portal 4 00000100:00000200:9.0:1547251627.674716:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445680, portal 4 00000100:00000200:9.0:1547251627.674719:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445680, offset 0 00000100:00000200:3.0:1547251627.674719:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445664, offset 0 00000400:00000200:9.0:1547251627.674722:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:3.0:1547251627.674723:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251627.674724:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:3.0:1547251627.674726:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251627.674727:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23049] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251627.674728:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23049] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251627.674730:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23048] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23049] 00000400:00000200:9.0:1547251627.674732:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:3.0:1547251627.674732:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23048] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23049] 00000400:00000200:9.0:1547251627.674734:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251627.674734:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251627.674735:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251627.674735:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251627.674736:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9747, 9748] 00000400:00000200:3.0:1547251627.674737:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251627.674738:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:3.0:1547251627.674738:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9747, 9748] 00000400:00000200:3.0:1547251627.674740:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:9.0:1547251627.674743:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251627.674745:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:9.0:1547251627.674748:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:3.0:1547251627.674749:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251627.674750:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:9.0:1547251627.674751:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:3.0:1547251627.674752:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:9.0:1547251627.674753:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:3.0:1547251627.674755:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251627.674756:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:3.0:1547251627.674758:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:3.0:1547251627.674760:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000100:00000200:14.0:1547251627.674766:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445696, portal 4 00000800:00000200:0.2:1547251627.674768:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000100:00000200:14.0:1547251627.674770:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445696, offset 0 00000400:00000200:14.0:1547251627.674774:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:12.2:1547251627.674776:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000400:00000200:14.0:1547251627.674777:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:14.0:1547251627.674780:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23050] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:14.0:1547251627.674783:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23049] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23050] 00000400:00000200:14.0:1547251627.674786:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:14.0:1547251627.674787:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251627.674789:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251627.674790:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9748, 9749] 00000400:00000200:14.0:1547251627.674792:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:14.0:1547251627.674796:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251627.674799:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:14.0:1547251627.674802:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251627.674809:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:14.0:1547251627.674810:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (23)++ 00000800:00000200:14.0:1547251627.674813:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:5.0:1547251627.674827:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)++ 00000800:00000200:5.0:1547251627.674831:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:4.0:1547251627.674831:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251627.674833:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000100:00000200:5.0:1547251627.674835:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf5100 x1622384012445680/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251627.674837:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b7800 x1622384012445664/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251627.674843:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea3b8 00000400:00000200:4.0:1547251627.674843:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555ee0 00000400:00000200:5.0:1547251627.674844:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251627.674846:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251627.674848:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251627.674849:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251627.674851:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251627.674853:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251627.674854:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251627.674854:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:5.0:1547251627.674855:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251627.674856:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:5.0:1547251627.674858:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f1e00 x1622384012445696/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251627.674863:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:5.0:1547251627.674865:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251627.674868:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251627.674869:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:0.2:1547251627.674923:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251627.674928:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:12.2:1547251627.674934:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:4.0:1547251627.674938:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251627.674951:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251627.674951:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.1.11.92@o2ib10 00000800:00000200:5.0:1547251627.674954:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251627.674956:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251627.674960:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251627.674961:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c047f0 00000400:00000200:5.0:1547251627.674964:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c047e0 00000400:00000200:4.0:1547251627.674966:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3d85 [1] + 192 00000400:00000200:5.0:1547251627.674971:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3d75 [1] + 192 00000100:00000200:4.0:1547251627.674975:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf5100 x1622384012445680/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251627.674987:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251627.674990:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b7800 x1622384012445664/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251627.674990:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251627.674992:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251627.674993:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251627.674995:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251627.674997:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251627.675000:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251627.675002:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf5100 x1622384012445680/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251627.675002:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000400:00000200:4.0:1547251627.675002:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:5.0:1547251627.675004:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251627.675005:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:4.0:1547251627.675005:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04800 00000800:00000200:5.0:1547251627.675007:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:9.0:1547251627.675008:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf5100 x1622384012445680/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251627.675009:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:5.0:1547251627.675010:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:9.0:1547251627.675017:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaaa0 00000400:00000200:4.0:1547251627.675020:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3d95 [1] + 192 00010000:00080000:9.0:1547251627.675021:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf5100 x1622384012445680/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251627.675025:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b7800 x1622384012445664/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251627.675031:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b7800 x1622384012445664/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251627.675035:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00000100:00000200:4.0:1547251627.675037:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f1e00 x1622384012445696/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:3.0:1547251627.675038:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b7800 x1622384012445664/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:4.0:1547251627.675045:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251627.675046:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251627.675048:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251627.675049:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000100:00000200:14.0:1547251627.675055:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f1e00 x1622384012445696/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:14.0:1547251627.675060:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f1e00 x1622384012445696/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251627.675064:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00010000:00080000:14.0:1547251627.675068:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f1e00 x1622384012445696/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251634 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251628.074685:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445712, portal 4 00000100:00000200:9.0:1547251628.074688:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445712, offset 0 00000400:00000200:9.0:1547251628.074690:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251628.074693:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251628.074696:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04810 00000400:00000200:9.0:1547251628.074698:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4839 [1] + 0 00000400:00000200:9.0:1547251628.074700:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d688110 00000100:00000200:9.0:1547251628.074702:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251628.074710:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688110 00000400:00000200:9.0:1547251628.074711:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251628.074713:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf4e00 x1622384012445712/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251635 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251628.074718:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea7f8 00000400:00000200:9.0:1547251628.074719:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251628.074720:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:11.0:1547251628.074743:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445712 00000100:00000200:6.0:1547251628.074749:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445728, portal 4 00000100:00000200:6.0:1547251628.074751:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445728, offset 0 00010000:00000200:11.0:1547251628.074754:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3da850 x1622384012445712/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:689/0 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:6.0:1547251628.074754:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251628.074757:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251628.074760:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04820 00000400:00000200:6.0:1547251628.074762:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2635 [1] + 0 00000100:00000200:11.0:1547251628.074764:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012445712, offset 192 00000400:00000200:6.0:1547251628.074764:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17eacc0 00000100:00000200:6.0:1547251628.074766:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:11.0:1547251628.074767:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:11.0:1547251628.074770:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:11.0:1547251628.074773:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04810 00000400:00000200:11.0:1547251628.074775:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf021 [1] + 192 00000400:00000200:6.0:1547251628.074775:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eacc0 00000400:00000200:6.0:1547251628.074776:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251628.074778:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf4e00 x1622384012445712/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251635 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251628.074782:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37933900 x1622384012445728/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251635 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251628.074788:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251628.074788:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990440 00000400:00000200:6.0:1547251628.074789:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251628.074790:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bd48 00000400:00000200:11.0:1547251628.074791:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251628.074791:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:11.0:1547251628.074792:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251628.074797:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf4e00 x1622384012445712/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251635 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251628.074798:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251628.074799:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445728 00000100:00000200:9.0:1547251628.074801:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf4e00 x1622384012445712/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251635 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251628.074805:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaaa0 00010000:00080000:9.0:1547251628.074808:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf4e00 x1622384012445712/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251635 ref 1 fl Rpc:R/0/0 rc 0/0 00010000:00000200:7.0:1547251628.074810:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39ac50 x1622384012445728/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:689/0 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:7.0:1547251628.074816:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012445728, offset 192 00000400:00000200:7.0:1547251628.074818:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251628.074821:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251628.074824:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04820 00000400:00000200:7.0:1547251628.074826:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf031 [1] + 192 00000100:00000200:7.0:1547251628.074828:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37933900 x1622384012445728/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251635 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251628.074837:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251628.074839:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea908 00000400:00000200:7.0:1547251628.074840:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251628.074843:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251628.074847:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37933900 x1622384012445728/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251635 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251628.074850:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251628.074853:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37933900 x1622384012445728/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251635 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251628.074859:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990330 00010000:00080000:6.0:1547251628.074862:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37933900 x1622384012445728/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251635 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251628.090682:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445744, portal 4 00000100:00000200:5.0:1547251628.090685:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445744, offset 0 00000400:00000200:5.0:1547251628.090688:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251628.090691:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251628.090693:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04830 00000400:00000200:5.0:1547251628.090696:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2665 [1] + 0 00000400:00000200:5.0:1547251628.090698:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17ea5d8 00000100:00000200:5.0:1547251628.090700:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:5.0:1547251628.090708:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea5d8 00000400:00000200:5.0:1547251628.090708:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251628.090711:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824449500 x1622384012445744/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251635 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251628.090716:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad5d8 00000400:00000200:5.0:1547251628.090716:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251628.090717:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251628.090738:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445744 00010000:00000200:7.0:1547251628.090747:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39a850 x1622384012445744/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:689/0 lens 224/368 e 0 to 0 dl 1547251634 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:7.0:1547251628.090754:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012445744, offset 192 00000400:00000200:7.0:1547251628.090756:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251628.090759:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251628.090762:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04830 00000400:00000200:7.0:1547251628.090765:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf041 [1] + 192 00000100:00000200:7.0:1547251628.090768:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824449500 x1622384012445744/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251635 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251628.090781:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251628.090783:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea110 00000400:00000200:7.0:1547251628.090784:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251628.090785:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251628.090788:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824449500 x1622384012445744/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251635 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251628.090790:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251628.090793:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824449500 x1622384012445744/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251635 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251628.090797:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad2a8 00010000:00080000:5.0:1547251628.090800:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824449500 x1622384012445744/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251635 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:12.2:1547251629.917358:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:0.2:1547251629.917371:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:4.0:1547251629.917415:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251629.917419:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251629.917420:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000800:00000200:5.0:1547251629.917423:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251629.917425:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251629.917427:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251629.917431:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b102e0 00000400:00000200:4.0:1547251629.917431:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b102d0 00000400:00000200:5.0:1547251629.917435:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 61376 00000400:00000200:4.0:1547251629.917439:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 61600 00000100:00000200:5.0:1547251629.917444:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:5.0:1547251629.917454:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251629.917454:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:5.0:1547251629.917457:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000400:00000200:4.0:1547251629.917459:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251629.917460:0:12662:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (23)++ 00000800:00000200:4.0:1547251629.917461:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251629.917463:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (24)-- 00000800:00000200:5.0:1547251629.917464:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:4.0:1547251629.917464:0:12661:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (23)++ 00000800:00000200:5.0:1547251629.917466:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (24)++ 00000800:00000200:4.0:1547251629.917466:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (24)-- 00000800:00000200:4.0:1547251629.917467:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (24)-- 00000800:00000200:5.0:1547251629.917469:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251629.917469:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251629.917471:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251629.917473:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251629.917475:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251629.917477:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 26 MB=0x5c38cb6b102c0 00000400:00000200:5.0:1547251629.917480:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0xd4fc9 [1] + 0 00000400:00000200:5.0:1547251629.917483:0:12662:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52dc38 00000100:00000200:5.0:1547251629.917485:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:5.0:1547251629.917493:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52dc38 00000400:00000200:5.0:1547251629.917494:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251629.917496:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (23)++ 00000800:00000200:5.0:1547251629.917498:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (24)-- 00000800:00000200:5.0:1547251629.917499:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251629.917501:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000100:00000200:4.0:1547251629.917502:0:13338:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444960 00000800:00000200:5.0:1547251629.917503:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000100:00000200:7.0:1547251629.917504:0:12981:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444944 00000800:00000200:5.0:1547251629.917504:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251629.917505:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (23)++ 00000800:00000200:5.0:1547251629.917506:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (24)-- 00000800:00000200:5.0:1547251629.917507:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (23)-- 00010000:00000200:4.0:1547251629.917508:0:13338:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae5c432700 x1622384011444960/t0(0) o400->cache1-MDT0000-lwp-OST0006_UUID@10.0.11.92@o2ib10:690/0 lens 224/224 e 0 to 0 dl 1547251635 ref 1 fl Interpret:H/0/0 rc 0/0 00000800:00000200:5.0:1547251629.917509:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:5.0:1547251629.917511:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00010000:00000200:7.0:1547251629.917514:0:12981:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444d400 x1622384011444944/t0(0) o400->cache1-MDT0000-lwp-OST0007_UUID@10.0.11.92@o2ib10:690/0 lens 224/224 e 0 to 0 dl 1547251635 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:0.0:1547251629.917515:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444928 00000100:00000200:4.0:1547251629.917517:0:13338:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444960, offset 192 00000400:00000200:4.0:1547251629.917521:0:13338:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00010000:00000200:0.0:1547251629.917521:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2d8050 x1622384011444928/t0(0) o400->04d65b15-cf9c-7ded-611d-ffe7fe3f3abf@10.0.11.92@o2ib10:690/0 lens 224/224 e 0 to 0 dl 1547251635 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:7.0:1547251629.917523:0:12981:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444944, offset 192 00000400:00000200:4.0:1547251629.917525:0:13338:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:7.0:1547251629.917527:0:12981:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000100:00000200:0.0:1547251629.917527:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384011444928, offset 192 00000400:00000200:4.0:1547251629.917528:0:13338:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251629.917530:0:13338:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251629.917530:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000400:00000200:4.0:1547251629.917532:0:13338:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9749, 9749] 00000400:00000200:0.0:1547251629.917534:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00000400:00000200:4.0:1547251629.917535:0:13338:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:0.0:1547251629.917536:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251629.917538:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251629.917539:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9749, 9750] 00000400:00000200:4.0:1547251629.917541:0:13338:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:0.0:1547251629.917541:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251629.917544:0:13338:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:7.0:1547251629.917546:0:12981:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00000400:00000200:7.0:1547251629.917548:0:12981:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251629.917550:0:12981:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251629.917551:0:12981:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9750, 9750] 00000400:00000200:7.0:1547251629.917553:0:12981:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:7.0:1547251629.917558:0:12981:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.1.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:7.0:1547251629.917561:0:12981:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:0.0:1547251629.917564:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.1.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251629.917567:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:4.0:1547251629.917568:0:13338:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:7.0:1547251629.917569:0:12981:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:7.0:1547251629.917570:0:12981:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:4.0:1547251629.917570:0:13338:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:7.0:1547251629.917571:0:12981:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251629.917571:0:13338:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:0.0:1547251629.917571:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251629.917572:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251629.917573:0:13338:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:7.0:1547251629.917574:0:12981:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:0.0:1547251629.917574:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (23)++ 00000800:00000200:0.0:1547251629.917577:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:0.2:1547251629.917579:0:12929:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (22)++ 00000400:00000200:4.0:1547251629.917581:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b102f0 00000800:00000200:12.2:1547251629.917592:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (23)++ 00000100:00000200:0.0:1547251629.917602:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:5.0:1547251629.917604:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (23)++ 00000400:00000200:4.0:1547251629.917606:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 61824 00000800:00000200:5.0:1547251629.917608:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (24)-- 00000100:00000200:4.0:1547251629.917609:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:4.0:1547251629.917616:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251629.917617:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b088 00000400:00000200:5.0:1547251629.917618:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251629.917618:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)++ 00000800:00000200:4.0:1547251629.917620:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (24)-- 00000400:00000200:5.0:1547251629.917621:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251629.917622:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:5.0:1547251629.917624:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251629.917625:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (23)++ 00000800:00000200:5.0:1547251629.917626:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251629.917626:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (24)-- 00000800:00000200:5.0:1547251629.917627:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (24)++ 00000800:00000200:5.0:1547251629.917628:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (24)-- 00000400:00000200:4.0:1547251629.917628:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eaaa0 00000400:00000200:4.0:1547251629.917629:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251629.917631:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665d48 00000400:00000200:5.0:1547251629.917632:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251629.917632:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251629.917634:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (23)-- 00000400:00000200:5.0:1547251629.917635:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251629.917636:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000100:00000200:7.0:1547251629.917637:0:12981:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444976 00000800:00000200:4.0:1547251629.917637:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00010000:00000200:7.0:1547251629.917642:0:12981:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444aa00 x1622384011444976/t0(0) o400->cache1-MDT0000-lwp-OST0008_UUID@10.0.11.92@o2ib10:690/0 lens 224/224 e 0 to 0 dl 1547251635 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:7.0:1547251629.917647:0:12981:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011444976, offset 192 00000400:00000200:7.0:1547251629.917650:0:12981:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:7.0:1547251629.917653:0:12981:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:7.0:1547251629.917655:0:12981:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251629.917656:0:12981:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251629.917657:0:12981:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9751, 9750] 00000400:00000200:7.0:1547251629.917659:0:12981:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251629.917665:0:12981:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:7.0:1547251629.917668:0:12981:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:7.0:1547251629.917671:0:12981:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:7.0:1547251629.917672:0:12981:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:7.0:1547251629.917673:0:12981:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:7.0:1547251629.917675:0:12981:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:0.2:1547251629.917688:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251629.917701:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:5.0:1547251629.917706:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000400:00000200:5.0:1547251629.917708:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea660 00000400:00000200:5.0:1547251629.917710:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251629.917713:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251629.917714:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251629.917715:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:9.2:1547251629.951702:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.2:1547251629.951717:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:14.0:1547251629.951756:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:14.0:1547251629.951760:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251629.951763:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251629.951767:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb35102e0 00000400:00000200:14.0:1547251629.951770:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 29696 00000100:00000200:14.0:1547251629.951773:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:12.0:1547251629.951779:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:14.0:1547251629.951783:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251629.951783:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000800:00000200:14.0:1547251629.951785:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:14.0:1547251629.951787:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:14.0:1547251629.951788:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:12.0:1547251629.951788:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:14.0:1547251629.951789:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:14.0:1547251629.951791:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251629.951795:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251629.951795:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb35102d0 00000400:00000200:14.0:1547251629.951797:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 26 MB=0x5c38cb35102c0 00000400:00000200:12.0:1547251629.951799:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 29920 00000400:00000200:14.0:1547251629.951800:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0xd4fe9 [1] + 0 00000400:00000200:14.0:1547251629.951802:0:12666:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d688f68 00000100:00000200:14.0:1547251629.951803:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000100:00000200:12.0:1547251629.951804:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:14.0:1547251629.951808:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688f68 00000400:00000200:14.0:1547251629.951809:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251629.951810:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:14.0:1547251629.951811:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:14.0:1547251629.951812:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:12.0:1547251629.951815:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251629.951816:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:14.0:1547251629.951818:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000800:00000200:12.0:1547251629.951818:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)++ 00000800:00000200:12.0:1547251629.951820:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (24)-- 00000400:00000200:14.0:1547251629.951822:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251629.951822:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251629.951824:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:13.0:1547251629.951825:0:12985:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821856 00000800:00000200:12.0:1547251629.951826:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:14.0:1547251629.951827:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb35102f0 00000400:00000200:14.0:1547251629.951830:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 30144 00010000:00000200:13.0:1547251629.951832:0:12985:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b0a8cab600 x1622383954821856/t0(0) o400->cache1-MDT0000-lwp-OST0009_UUID@10.0.11.93@o2ib10:690/0 lens 224/224 e 0 to 0 dl 1547251635 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:14.0:1547251629.951833:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:12.0:1547251629.951835:0:13371:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821840 00000400:00000200:14.0:1547251629.951838:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251629.951839:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00010000:00000200:12.0:1547251629.951840:0:13371:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97bae1af4800 x1622383954821840/t0(0) o400->cache1-MDT0000-lwp-OST000a_UUID@10.0.11.93@o2ib10:690/0 lens 224/224 e 0 to 0 dl 1547251635 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:13.0:1547251629.951841:0:12985:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821856, offset 192 00000800:00000200:14.0:1547251629.951842:0:12666:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:14.0:1547251629.951845:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000400:00000200:13.0:1547251629.951845:0:12985:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:14.0:1547251629.951846:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:12.0:1547251629.951847:0:13371:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821840, offset 192 00000400:00000200:13.0:1547251629.951849:0:12985:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:12.0:1547251629.951850:0:13371:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000400:00000200:13.0:1547251629.951852:0:12985:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:14.0:1547251629.951853:0:13337:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821872 00000400:00000200:13.0:1547251629.951854:0:12985:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251629.951856:0:12985:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9750, 9749] 00000800:00000200:7.2:1547251629.951857:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00010000:00000200:14.0:1547251629.951859:0:13337:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b0a8ca8000 x1622383954821872/t0(0) o400->cache1-MDT0000-lwp-OST000b_UUID@10.0.11.93@o2ib10:690/0 lens 224/224 e 0 to 0 dl 1547251635 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:13.0:1547251629.951859:0:12985:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:13.0:1547251629.951865:0:12985:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000100:00000200:0.0:1547251629.951867:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821824 00000100:00000200:14.0:1547251629.951868:0:13337:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821872, offset 192 00000800:00000200:13.0:1547251629.951869:0:12985:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:12.0:1547251629.951870:0:13371:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00000400:00000200:14.0:1547251629.951872:0:13337:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000400:00000200:12.0:1547251629.951873:0:13371:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00010000:00000200:0.0:1547251629.951873:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2dd050 x1622383954821824/t0(0) o400->b99b3dbb-ff3e-ba67-561d-3f0ab43da035@10.0.11.93@o2ib10:690/0 lens 224/224 e 0 to 0 dl 1547251635 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:12.0:1547251629.951874:0:13371:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251629.951876:0:13371:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9750, 9750] 00000400:00000200:12.0:1547251629.951878:0:13371:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000100:00000200:0.0:1547251629.951879:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954821824, offset 192 00000400:00000200:0.0:1547251629.951882:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:12.0:1547251629.951884:0:13371:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.1.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:0.0:1547251629.951886:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000800:00000200:12.0:1547251629.951888:0:13371:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:0.0:1547251629.951888:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251629.951889:0:13337:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00000800:00000200:13.0:1547251629.951889:0:12985:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000400:00000200:0.0:1547251629.951890:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:13.0:1547251629.951891:0:12985:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000400:00000200:0.0:1547251629.951891:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9751, 9750] 00000400:00000200:14.0:1547251629.951892:0:13337:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:13.0:1547251629.951892:0:12985:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:14.0:1547251629.951893:0:13337:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251629.951893:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251629.951894:0:13337:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9751, 9750] 00000800:00000200:13.0:1547251629.951895:0:12985:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:14.0:1547251629.951896:0:13337:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251629.951902:0:13337:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.1.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251629.951905:0:13337:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:12.0:1547251629.951907:0:13371:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251629.951909:0:13371:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:9.2:1547251629.951909:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251629.951910:0:13371:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (23)++ 00000400:00000200:0.0:1547251629.951910:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251629.951913:0:13337:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251629.951913:0:13371:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (24)-- 00000800:00000200:0.0:1547251629.951913:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:14.0:1547251629.951915:0:13337:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:14.0:1547251629.951915:0:13337:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:0.0:1547251629.951916:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251629.951917:0:13337:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:0.0:1547251629.951917:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:0.0:1547251629.951918:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:12.0:1547251629.951921:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)++ 00000800:00000200:0.0:1547251629.951921:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (24)-- 00000800:00000200:12.0:1547251629.951923:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (24)-- 00000800:00000200:12.0:1547251629.951924:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:0.0:1547251629.951925:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:12.0:1547251629.951926:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:14.0:1547251629.951927:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251629.951927:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (24)-- 00000800:00000200:14.0:1547251629.951928:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000400:00000200:12.0:1547251629.951929:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b3b8 00000800:00000200:7.2:1547251629.951929:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000400:00000200:12.0:1547251629.951930:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251629.951931:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b0383b8 00000400:00000200:14.0:1547251629.951932:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251629.951934:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:12.0:1547251629.951934:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251629.951938:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:14.0:1547251629.951939:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251629.951940:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:14.0:1547251629.951950:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:12.0:1547251629.951951:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:14.0:1547251629.951952:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:14.0:1547251629.951953:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000400:00000200:12.0:1547251629.951953:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00000400:00000200:14.0:1547251629.951954:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00000400:00000200:12.0:1547251629.951954:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251629.951955:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251629.951957:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:12.0:1547251629.951957:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251629.951958:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251629.951959:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:14.0:1547251629.951961:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:12.0:1547251629.951962:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:7.2:1547251629.983453:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:5.2:1547251629.983468:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:10.0:1547251629.983515:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251629.983518:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251629.983519:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251629.983522:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251629.983524:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251629.983526:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251629.983529:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510320 00000400:00000200:11.0:1547251629.983530:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510310 00000400:00000200:10.0:1547251629.983533:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 13440 00000400:00000200:11.0:1547251629.983537:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 13664 00000100:00000200:10.0:1547251629.983540:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:11.0:1547251629.983552:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:10.0:1547251629.983552:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251629.983554:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251629.983556:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000400:00000200:11.0:1547251629.983557:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251629.983557:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251629.983559:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:10.0:1547251629.983559:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251629.983561:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251629.983561:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251629.983563:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251629.983564:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000400:00000200:10.0:1547251629.983566:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251629.983567:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251629.983569:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510300 00000400:00000200:11.0:1547251629.983571:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251629.983572:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 13888 00000100:00000200:10.0:1547251629.983575:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:10.0:1547251629.983578:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251629.983579:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251629.983580:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251629.983582:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251629.983595:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:10.0:1547251629.983597:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000100:00000200:10.0:1547251629.983602:0:13336:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821920 00000100:00000200:11.0:1547251629.983605:0:12983:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821888 00000100:00000200:9.0:1547251629.983607:0:12982:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821904 00010000:00000200:10.0:1547251629.983609:0:13336:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b17b078c00 x1622383954821920/t0(0) o400->cache1-MDT0000-lwp-OST0004_UUID@10.0.11.91@o2ib10:690/0 lens 224/224 e 0 to 0 dl 1547251635 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:11.0:1547251629.983610:0:12983:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b17b07aa00 x1622383954821888/t0(0) o400->cache1-MDT0000-lwp-OST0005_UUID@10.0.11.91@o2ib10:690/0 lens 224/224 e 0 to 0 dl 1547251635 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:9.0:1547251629.983615:0:12982:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ba960e9800 x1622383954821904/t0(0) o400->cache1-MDT0000-lwp-OST0003_UUID@10.0.11.91@o2ib10:690/0 lens 224/224 e 0 to 0 dl 1547251635 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:11.0:1547251629.983617:0:12983:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821888, offset 192 00000100:00000200:10.0:1547251629.983618:0:13336:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821920, offset 192 00000400:00000200:10.0:1547251629.983621:0:13336:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:11.0:1547251629.983623:0:12983:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:10.0:1547251629.983625:0:13336:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000100:00000200:9.0:1547251629.983625:0:12982:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821904, offset 192 00000400:00000200:10.0:1547251629.983628:0:13336:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251629.983629:0:12982:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000400:00000200:10.0:1547251629.983630:0:13336:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251629.983631:0:13336:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9755, 9756] 00000400:00000200:10.0:1547251629.983633:0:13336:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251629.983640:0:13336:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:10.0:1547251629.983644:0:13336:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:11.0:1547251629.983645:0:12983:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:11.0:1547251629.983647:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251629.983649:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251629.983651:0:12983:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9756, 9756] 00000400:00000200:11.0:1547251629.983653:0:12983:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251629.983659:0:12983:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251629.983662:0:12983:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251629.983664:0:12982:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251629.983667:0:12982:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251629.983669:0:12982:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251629.983670:0:12982:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9756, 9757] 00000400:00000200:9.0:1547251629.983672:0:12982:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:9.0:1547251629.983677:0:12982:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.91@o2ib10(10.1.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251629.983680:0:12982:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:11.0:1547251629.983681:0:12983:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:10.0:1547251629.983681:0:13336:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251629.983683:0:12983:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:10.0:1547251629.983683:0:13336:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:9.0:1547251629.983683:0:12982:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251629.983684:0:12983:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251629.983684:0:13336:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:9.0:1547251629.983684:0:12982:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:9.0:1547251629.983685:0:12982:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251629.983687:0:12983:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251629.983687:0:13336:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000800:00000200:9.0:1547251629.983688:0:12982:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (23)-- 00000400:00000200:11.0:1547251629.983697:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 26 MB=0x5c38cb35102f0 00000800:00000200:7.2:1547251629.983699:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000400:00000200:11.0:1547251629.983701:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0xd5079 [1] + 0 00000400:00000200:11.0:1547251629.983704:0:12663:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d688dd0 00000100:00000200:11.0:1547251629.983706:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000800:00000200:5.2:1547251629.983706:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (22)++ 00000800:00000200:7.2:1547251629.983708:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:11.0:1547251629.983714:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688dd0 00000400:00000200:11.0:1547251629.983715:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251629.983717:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (23)++ 00000800:00000200:10.0:1547251629.983718:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251629.983719:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (24)-- 00000800:00000200:10.0:1547251629.983719:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251629.983720:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (23)-- 00000400:00000200:10.0:1547251629.983721:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914dd0 00000800:00000200:11.0:1547251629.983722:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000400:00000200:10.0:1547251629.983722:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251629.983724:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000400:00000200:10.0:1547251629.983725:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251629.983726:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaaa0 00000400:00000200:11.0:1547251629.983727:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251629.983727:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251629.983729:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000400:00000200:11.0:1547251629.983730:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251629.983730:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000400:00000200:10.0:1547251629.983731:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b2a8 00000800:00000200:11.0:1547251629.983732:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:10.0:1547251629.983732:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251629.983734:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000400:00000200:10.0:1547251629.983735:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251629.983736:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:10.0:1547251629.983737:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251629.983738:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:0.0:1547251629.983738:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821872 00010000:00000200:0.0:1547251629.983744:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2dbc50 x1622383954821872/t0(0) o400->dc7951f7-3e71-e9c4-2e5e-5f95f4837d68@10.0.11.91@o2ib10:690/0 lens 224/224 e 0 to 0 dl 1547251635 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:0.0:1547251629.983751:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954821872, offset 192 00000400:00000200:0.0:1547251629.983754:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000400:00000200:0.0:1547251629.983757:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000400:00000200:0.0:1547251629.983759:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251629.983761:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251629.983762:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9757, 9757] 00000400:00000200:0.0:1547251629.983764:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:0.0:1547251629.983769:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.1.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251629.983772:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:0.0:1547251629.983775:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251629.983776:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:0.0:1547251629.983778:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:0.0:1547251629.983780:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000100:00000200:0.0:1547251629.983784:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:5.2:1547251629.983793:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251629.983805:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251629.983811:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000400:00000200:10.0:1547251629.983813:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665440 00000400:00000200:10.0:1547251629.983815:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251629.983818:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251629.983818:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251629.983821:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:10.0:1547251630.106677:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445760, portal 4 00000100:00000200:10.0:1547251630.106680:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445760, offset 0 00000400:00000200:10.0:1547251630.106683:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:10.0:1547251630.106686:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:10.0:1547251630.106688:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23050] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:10.0:1547251630.106691:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23050] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23050] 00000400:00000200:10.0:1547251630.106693:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:10.0:1547251630.106695:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251630.106696:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251630.106697:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9751, 9752] 00000100:00000200:6.0:1547251630.106698:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445776, portal 4 00000400:00000200:10.0:1547251630.106699:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000100:00000200:3.0:1547251630.106700:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445792, portal 4 00000100:00000200:6.0:1547251630.106701:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445776, offset 0 00000100:00000200:3.0:1547251630.106703:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445792, offset 0 00000400:00000200:10.0:1547251630.106704:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:6.0:1547251630.106704:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:10.0:1547251630.106707:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:6.0:1547251630.106707:0:12676:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:3.0:1547251630.106707:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:6.0:1547251630.106709:0:12676:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23051] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:10.0:1547251630.106710:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000400:00000200:3.0:1547251630.106710:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000800:00000200:10.0:1547251630.106711:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000400:00000200:3.0:1547251630.106712:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23051] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:10.0:1547251630.106713:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:6.0:1547251630.106713:0:12676:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23050] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23051] 00000400:00000200:6.0:1547251630.106715:0:12676:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:10.0:1547251630.106716:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:3.0:1547251630.106716:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23050] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23051] 00000400:00000200:6.0:1547251630.106717:0:12676:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251630.106718:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:6.0:1547251630.106719:0:12676:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251630.106720:0:12676:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9752, 9752] 00000400:00000200:3.0:1547251630.106720:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251630.106721:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251630.106722:0:12676:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:3.0:1547251630.106723:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9752, 9752] 00000400:00000200:3.0:1547251630.106724:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:6.0:1547251630.106730:0:12676:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:7.2:1547251630.106734:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:6.0:1547251630.106734:0:12676:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:3.0:1547251630.106736:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:6.0:1547251630.106738:0:12676:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251630.106739:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:6.0:1547251630.106740:0:12676:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:6.0:1547251630.106742:0:12676:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:3.0:1547251630.106742:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251630.106744:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:6.0:1547251630.106745:0:12676:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:3.0:1547251630.106745:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:3.0:1547251630.106748:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:9.2:1547251630.106764:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:14.0:1547251630.106787:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:14.0:1547251630.106790:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:14.0:1547251630.106794:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960ead00 x1622384012445760/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251637 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251630.106799:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914990 00000400:00000200:14.0:1547251630.106801:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251630.106803:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251630.106806:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:14.0:1547251630.106807:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:14.0:1547251630.106809:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (24)-- 00000100:00000200:14.0:1547251630.106811:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37936900 x1622384012445776/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251637 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251630.106815:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990440 00000400:00000200:14.0:1547251630.106816:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251630.106818:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251630.106820:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:14.0:1547251630.106821:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:14.0:1547251630.106822:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:14.0:1547251630.106823:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:14.0:1547251630.106825:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c3600 x1622384012445792/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251637 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251630.106825:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:14.0:1547251630.106829:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555550 00000400:00000200:14.0:1547251630.106830:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:14.0:1547251630.106832:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:14.0:1547251630.106833:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:9.2:1547251630.106861:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:12.0:1547251630.106873:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251630.106883:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251630.106884:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251630.106886:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:12.0:1547251630.106886:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251630.106888:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:7.2:1547251630.106900:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251630.106906:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:9.2:1547251630.106910:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:14.0:1547251630.106914:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:14.0:1547251630.106917:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251630.106917:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251630.106919:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251630.106920:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251630.106923:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251630.106924:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04850 00000400:00000200:14.0:1547251630.106927:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fced [1] + 192 00000400:00000200:12.0:1547251630.106927:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04840 00000400:00000200:12.0:1547251630.106931:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fcdd [1] + 192 00000100:00000200:14.0:1547251630.106935:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37936900 x1622384012445776/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251637 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251630.106952:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251630.106954:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000100:00000200:12.0:1547251630.106954:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960ead00 x1622384012445760/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251637 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251630.106956:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:14.0:1547251630.106957:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:14.0:1547251630.106959:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:14.0:1547251630.106961:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251630.106964:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251630.106964:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251630.106966:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000100:00000200:6.0:1547251630.106966:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37936900 x1622384012445776/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251637 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251630.106967:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04860 00000800:00000200:12.0:1547251630.106968:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251630.106969:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251630.106971:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:12.0:1547251630.106972:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:12.0:1547251630.106973:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:6.0:1547251630.106977:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37936900 x1622384012445776/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251637 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251630.106981:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990330 00000400:00000200:14.0:1547251630.106983:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fcfd [1] + 192 00010000:00080000:6.0:1547251630.106985:0:12676:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ae37936900 x1622384012445776/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251637 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251630.106987:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960ead00 x1622384012445760/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251637 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:10.0:1547251630.106993:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960ead00 x1622384012445760/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251637 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251630.106997:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914dd0 00000100:00000200:14.0:1547251630.106999:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c3600 x1622384012445792/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251637 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:10.0:1547251630.107000:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba960ead00 x1622384012445760/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251637 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:14.0:1547251630.107006:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251630.107008:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:14.0:1547251630.107009:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:14.0:1547251630.107010:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:3.0:1547251630.107019:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c3600 x1622384012445792/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251637 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251630.107025:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c3600 x1622384012445792/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251637 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251630.107029:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251630.107033:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c3600 x1622384012445792/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251637 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251631.610705:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012445856, portal 10 00000100:00000200:4.0:1547251631.610705:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012445824, portal 10 00000100:00000200:5.0:1547251631.610709:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012445856, offset 0 00000100:00000200:4.0:1547251631.610709:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012445824, offset 0 00000400:00000200:5.0:1547251631.610712:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251631.610716:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251631.610716:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251631.610719:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c048a0 00000400:00000200:4.0:1547251631.610719:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251631.610722:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c04880 00000400:00000200:5.0:1547251631.610723:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 62048 00000400:00000200:4.0:1547251631.610726:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 62272 00000100:00000200:5.0:1547251631.610730:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:7.0:1547251631.610734:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012445840, portal 10 00000100:00000200:6.0:1547251631.610734:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012445808, portal 25 00000400:00000200:5.0:1547251631.610734:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251631.610735:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:4.0:1547251631.610739:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251631.610740:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c436900 x1622384012445856/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251631.610747:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad5d8 00000100:00000200:7.0:1547251631.610748:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012445840, offset 0 00000400:00000200:5.0:1547251631.610748:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:6.0:1547251631.610749:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 26, xid 1622384012445808, offset 0 00000400:00000200:5.0:1547251631.610749:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251631.610749:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c432400 x1622384012445824/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251631.610756:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bbb0 00000400:00000200:4.0:1547251631.610757:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251631.610758:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251631.610758:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251631.610761:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251631.610762:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251631.610764:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251631.610766:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c04890 00000400:00000200:6.0:1547251631.610767:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 26 MB=0x5c38cb6c04870 00000400:00000200:7.0:1547251631.610770:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 62496 00000400:00000200:6.0:1547251631.610770:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-0@lo of length 224/224 into md 0xd5259 [1] + 0 00000100:00000200:5.0:1547251631.610771:0:13338:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445856 00000100:00000200:7.0:1547251631.610773:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:6.0:1547251631.610773:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d688aa0 00000100:00000200:4.0:1547251631.610774:0:12981:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445824 00000100:00000200:6.0:1547251631.610775:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00010000:00000200:5.0:1547251631.610776:0:13338:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae5c435700 x1622384012445856/t0(0) o400->cache1-MDT0000-lwp-OST0002_UUID@0@lo:692/0 lens 224/224 e 0 to 0 dl 1547251637 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:4.0:1547251631.610777:0:12981:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444ce00 x1622384012445824/t0(0) o400->cache1-MDT0000-lwp-MDT0000_UUID@0@lo:692/0 lens 224/224 e 0 to 0 dl 1547251637 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:7.0:1547251631.610779:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251631.610783:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c436300 x1622384012445840/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251631.610785:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688aa0 00000400:00000200:6.0:1547251631.610786:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251631.610789:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eaee0 00000400:00000200:7.0:1547251631.610790:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:4.0:1547251631.610791:0:12981:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012445824, offset 192 00000400:00000200:7.0:1547251631.610792:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251631.610792:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c434b00 x1622384012445808/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251638 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251631.610798:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459903b8 00000400:00000200:6.0:1547251631.610799:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:5.0:1547251631.610800:0:13338:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012445856, offset 192 00000400:00000200:6.0:1547251631.610801:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251631.610801:0:12981:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251631.610802:0:13338:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251631.610804:0:12981:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251631.610805:0:13338:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:6.0:1547251631.610806:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012445872, portal 10 00000400:00000200:4.0:1547251631.610807:0:12981:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c04880 00000400:00000200:5.0:1547251631.610808:0:13338:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c048a0 00000100:00000200:6.0:1547251631.610809:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012445872, offset 0 00000400:00000200:5.0:1547251631.610810:0:13338:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaf091 [1] + 192 00000100:00000200:7.0:1547251631.610811:0:12979:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445840 00000400:00000200:6.0:1547251631.610811:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251631.610813:0:12981:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaf0a1 [1] + 192 00000400:00000200:6.0:1547251631.610814:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00010000:00000200:7.0:1547251631.610816:0:12979:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae37931500 x1622384012445840/t0(0) o400->cache1-MDT0000-lwp-OST0000_UUID@0@lo:692/0 lens 224/224 e 0 to 0 dl 1547251637 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:5.0:1547251631.610816:0:13338:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c436900 x1622384012445856/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251631.610817:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c048b0 00000400:00000200:6.0:1547251631.610820:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 62720 00000400:00000200:5.0:1547251631.610822:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251631.610823:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:4.0:1547251631.610823:0:12981:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c432400 x1622384012445824/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251631.610826:0:12979:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012445840, offset 192 00000400:00000200:6.0:1547251631.610826:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251631.610826:0:13338:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad5d8 00000100:00000200:6.0:1547251631.610828:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c435d00 x1622384012445872/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251631.610828:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251631.610829:0:13338:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251631.610829:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251631.610833:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990220 00000400:00000200:6.0:1547251631.610836:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251631.610837:0:12979:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251631.610837:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251631.610838:0:12981:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bbb0 00000400:00000200:7.0:1547251631.610839:0:12979:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:5.0:1547251631.610839:0:13338:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445872 00000400:00000200:4.0:1547251631.610839:0:12981:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251631.610841:0:12981:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251631.610842:0:12979:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c04890 00010000:00000200:5.0:1547251631.610842:0:13338:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a765df6f00 x1622384012445872/t0(0) o400->cache1-MDT0000-lwp-OST0001_UUID@0@lo:692/0 lens 224/224 e 0 to 0 dl 1547251637 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:2.0:1547251631.610844:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445808 00000400:00000200:7.0:1547251631.610845:0:12979:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaf0b1 [1] + 192 00000100:00000200:7.0:1547251631.610848:0:12979:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c436300 x1622384012445840/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251631.610849:0:13338:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012445872, offset 192 00010000:00000200:2.0:1547251631.610849:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a48f2df050 x1622384012445808/t0(0) o400->a65be517-33e7-fa36-0de3-da9e02df4f8d@0@lo:692/0 lens 224/224 e 0 to 0 dl 1547251637 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:5.0:1547251631.610851:0:13338:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251631.610853:0:13338:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251631.610855:0:12979:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251631.610856:0:13338:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c048b0 00000100:00000200:4.0:1547251631.610857:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c432400 x1622384012445824/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251631.610858:0:12979:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eaee0 00000400:00000200:7.0:1547251631.610859:0:12979:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251631.610861:0:12979:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251631.610862:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c432400 x1622384012445824/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251631.610867:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b330 00000100:00000200:2.0:1547251631.610869:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384012445808, offset 192 00000400:00000200:5.0:1547251631.610870:0:13338:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaf0d1 [1] + 192 00000100:00000200:6.0:1547251631.610874:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c436300 x1622384012445840/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251631.610880:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c436300 x1622384012445840/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251631.610885:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eaf68 00000400:00000200:2.0:1547251631.610888:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:5.0:1547251631.610890:0:13338:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c435d00 x1622384012445872/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251631.610891:0:12929:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:2.0:1547251631.610893:0:12929:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 25 MB=0x5c38cb6c04870 00000400:00000200:5.0:1547251631.610902:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:2.0:1547251631.610903:0:12929:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 19 from 12345-0@lo of length 192/192 into md 0xaf0c1 [1] + 192 00000400:00000200:5.0:1547251631.610904:0:13338:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad5d8 00000400:00000200:5.0:1547251631.610905:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:2.0:1547251631.610906:0:12929:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c434b00 x1622384012445808/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251631.610907:0:13338:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:2.0:1547251631.610911:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251631.610913:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c435d00 x1622384012445872/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251631.610917:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c435d00 x1622384012445872/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251631.610921:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459903b8 00000400:00000200:2.0:1547251631.610922:0:12929:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a088 00000400:00000200:2.0:1547251631.610923:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:5.0:1547251631.610924:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c436900 x1622384012445856/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251631.610924:0:12929:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251631.610928:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c436900 x1622384012445856/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251631.610929:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000400:00000200:5.0:1547251631.610932:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad2a8 00000100:00000200:6.0:1547251631.610934:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c434b00 x1622384012445808/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251631.610938:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c434b00 x1622384012445808/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251638 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251631.610949:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990330 00000100:00000200:11.0:1547251632.218715:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445888, portal 4 00000100:00000200:11.0:1547251632.218719:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445888, offset 0 00000400:00000200:11.0:1547251632.218723:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:11.0:1547251632.218725:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:11.0:1547251632.218728:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23051] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:11.0:1547251632.218731:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23052] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23051] 00000400:00000200:11.0:1547251632.218733:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:11.0:1547251632.218734:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251632.218736:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251632.218737:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9757, 9758] 00000400:00000200:11.0:1547251632.218739:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251632.218743:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251632.218746:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:11.0:1547251632.218749:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000100:00000200:15.0:1547251632.218750:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445904, portal 4 00000800:00000200:11.0:1547251632.218750:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:11.0:1547251632.218751:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251632.218754:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)-- 00000100:00000200:15.0:1547251632.218755:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445904, offset 0 00000400:00000200:15.0:1547251632.218760:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:15.0:1547251632.218763:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:15.0:1547251632.218766:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23052] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:2.0:1547251632.218768:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445920, portal 4 00000400:00000200:15.0:1547251632.218769:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23052] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23052] 00000400:00000200:15.0:1547251632.218771:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000100:00000200:2.0:1547251632.218771:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445920, offset 0 00000400:00000200:15.0:1547251632.218772:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251632.218774:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251632.218775:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9758, 9758] 00000400:00000200:2.0:1547251632.218776:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:15.0:1547251632.218777:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:2.0:1547251632.218779:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:15.0:1547251632.218782:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000400:00000200:2.0:1547251632.218782:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23052] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:15.0:1547251632.218785:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:2.0:1547251632.218785:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23053] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23052] 00000400:00000200:2.0:1547251632.218787:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:15.0:1547251632.218788:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251632.218789:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000400:00000200:2.0:1547251632.218789:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:15.0:1547251632.218790:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:2.0:1547251632.218791:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251632.218792:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9758, 9759] 00000800:00000200:15.0:1547251632.218793:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:2.0:1547251632.218794:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:2.0:1547251632.218798:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251632.218801:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:2.0:1547251632.218804:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:5.2:1547251632.218805:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00000800:00000200:2.0:1547251632.218806:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:2.0:1547251632.218808:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (23)++ 00000800:00000200:2.0:1547251632.218811:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (24)-- 00000800:00000200:11.0:1547251632.218822:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (23)++ 00000800:00000200:11.0:1547251632.218830:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (24)-- 00000100:00000200:11.0:1547251632.218834:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf4b00 x1622384012445888/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251632.218838:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (23)++ 00000800:00000200:10.0:1547251632.218840:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (24)-- 00000400:00000200:11.0:1547251632.218841:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b908 00000400:00000200:11.0:1547251632.218843:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:10.0:1547251632.218843:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328de300 x1622384012445920/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:7.2:1547251632.218844:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:11.0:1547251632.218846:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251632.218850:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (23)-- 00000400:00000200:10.0:1547251632.218850:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06aaa0 00000400:00000200:10.0:1547251632.218851:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251632.218852:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251632.218854:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000400:00000200:10.0:1547251632.218854:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251632.218856:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251632.218856:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:10.0:1547251632.218858:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000100:00000200:11.0:1547251632.218859:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af2d00 x1622384012445904/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251632.218864:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1bb0 00000400:00000200:11.0:1547251632.218866:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251632.218869:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251632.218871:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:5.2:1547251632.218933:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251632.218940:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:10.0:1547251632.218955:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251632.218959:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251632.218962:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000400:00000200:10.0:1547251632.218963:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251632.218964:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251632.218968:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c048d0 00000400:00000200:11.0:1547251632.218969:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251632.218972:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7049 [1] + 192 00000400:00000200:11.0:1547251632.218973:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c048c0 00000400:00000200:11.0:1547251632.218976:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7029 [1] + 192 00000100:00000200:10.0:1547251632.218981:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af2d00 x1622384012445904/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251632.218993:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251632.218994:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf4b00 x1622384012445888/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251632.218995:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251632.218997:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:10.0:1547251632.218998:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251632.219000:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:10.0:1547251632.219002:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251632.219004:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251632.219005:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:15.0:1547251632.219007:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af2d00 x1622384012445904/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251632.219007:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (23)++ 00000800:00000200:11.0:1547251632.219009:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (24)-- 00000800:00000200:11.0:1547251632.219010:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000400:00000200:10.0:1547251632.219010:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251632.219012:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000100:00000200:15.0:1547251632.219013:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af2d00 x1622384012445904/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251632.219013:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c048e0 00000400:00000200:15.0:1547251632.219017:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb15d8 00000400:00000200:10.0:1547251632.219020:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7059 [1] + 192 00010000:00080000:15.0:1547251632.219021:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af2d00 x1622384012445904/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251632.219024:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328de300 x1622384012445920/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251632.219035:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251632.219037:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251632.219038:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251632.219039:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:2.0:1547251632.219044:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328de300 x1622384012445920/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251632.219050:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328de300 x1622384012445920/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251632.219054:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06ab28 00010000:00080000:2.0:1547251632.219058:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328de300 x1622384012445920/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251632.219059:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf4b00 x1622384012445888/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251632.219065:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf4b00 x1622384012445888/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251632.219070:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b770 00010000:00080000:8.0:1547251632.219073:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf4b00 x1622384012445888/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251632.682680:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445936, portal 4 00000100:00000200:8.0:1547251632.682683:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445936, offset 0 00000400:00000200:8.0:1547251632.682686:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251632.682688:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251632.682691:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23053] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251632.682694:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23053] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23053] 00000400:00000200:8.0:1547251632.682696:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251632.682698:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251632.682699:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251632.682700:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9751, 9751] 00000400:00000200:8.0:1547251632.682702:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000100:00000200:15.0:1547251632.682704:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445952, portal 4 00000400:00000200:8.0:1547251632.682706:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000100:00000200:15.0:1547251632.682707:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445952, offset 0 00000800:00000200:8.0:1547251632.682709:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:15.0:1547251632.682711:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:8.0:1547251632.682712:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:15.0:1547251632.682713:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:8.0:1547251632.682714:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000400:00000200:15.0:1547251632.682716:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23054] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:8.0:1547251632.682716:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:8.0:1547251632.682718:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000400:00000200:15.0:1547251632.682719:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23053] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23054] 00000400:00000200:15.0:1547251632.682721:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:15.0:1547251632.682724:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251632.682725:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251632.682726:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9751, 9752] 00000400:00000200:15.0:1547251632.682728:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:15.0:1547251632.682733:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251632.682735:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:15.0:1547251632.682738:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251632.682740:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251632.682740:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:15.0:1547251632.682741:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:15.0:1547251632.682744:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:0.2:1547251632.682755:0:13057:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000100:00000200:3.0:1547251632.682762:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445968, portal 4 00000100:00000200:3.0:1547251632.682766:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445968, offset 0 00000400:00000200:3.0:1547251632.682770:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:3.0:1547251632.682774:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:3.0:1547251632.682777:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23054] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251632.682781:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23054] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23054] 00000400:00000200:3.0:1547251632.682785:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251632.682787:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251632.682790:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251632.682791:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9752, 9752] 00000400:00000200:3.0:1547251632.682793:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:3.0:1547251632.682798:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251632.682801:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:4.0:1547251632.682802:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:3.0:1547251632.682804:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251632.682805:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (23)++ 00000800:00000200:5.0:1547251632.682806:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251632.682806:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:3.0:1547251632.682806:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (24)++ 00000800:00000200:3.0:1547251632.682809:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (24)-- 00000800:00000200:5.0:1547251632.682810:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:4.0:1547251632.682810:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824751200 x1622384012445936/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251632.682813:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8ca8c00 x1622384012445952/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251632.682818:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000400:00000200:4.0:1547251632.682819:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251632.682820:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1dd0 00000400:00000200:5.0:1547251632.682821:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251632.682823:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:5.0:1547251632.682824:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251632.682826:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251632.682827:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251632.682828:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251632.682829:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:4.0:1547251632.682830:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251632.682831:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:4.0:1547251632.682832:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b7500 x1622384012445968/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251632.682838:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555088 00000400:00000200:4.0:1547251632.682839:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251632.682842:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251632.682844:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:12.2:1547251632.682903:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:0.2:1547251632.682908:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:5.0:1547251632.682918:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251632.682921:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251632.682922:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251632.682925:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251632.682926:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251632.682929:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251632.682931:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04900 00000400:00000200:4.0:1547251632.682933:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c048f0 00000400:00000200:5.0:1547251632.682935:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3ee5 [1] + 192 00000400:00000200:4.0:1547251632.682940:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3ed5 [1] + 192 00000100:00000200:5.0:1547251632.682954:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8ca8c00 x1622384012445952/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251632.682967:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251632.682969:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000100:00000200:4.0:1547251632.682969:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824751200 x1622384012445936/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251632.682972:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251632.682973:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251632.682975:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:5.0:1547251632.682976:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:5.0:1547251632.682979:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251632.682980:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:15.0:1547251632.682982:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8ca8c00 x1622384012445952/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251632.682982:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)++ 00000400:00000200:5.0:1547251632.682983:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251632.682984:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (24)-- 00000400:00000200:5.0:1547251632.682986:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04910 00000800:00000200:4.0:1547251632.682986:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (23)-- 00000100:00000200:15.0:1547251632.682988:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8ca8c00 x1622384012445952/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251632.682988:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:15.0:1547251632.682992:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb15d8 00010000:00080000:15.0:1547251632.682996:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8ca8c00 x1622384012445952/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251632.682996:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3ef5 [1] + 192 00000100:00000200:8.0:1547251632.683001:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824751200 x1622384012445936/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251632.683006:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824751200 x1622384012445936/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251632.683010:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000100:00000200:5.0:1547251632.683013:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b7500 x1622384012445968/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251632.683014:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824751200 x1622384012445936/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251632.683021:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251632.683022:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251632.683023:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251632.683024:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000100:00000200:3.0:1547251632.683030:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b7500 x1622384012445968/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251632.683035:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b7500 x1622384012445968/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251632.683040:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251632.683043:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b7500 x1622384012445968/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251639 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251633.082681:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012445984, portal 4 00000100:00000200:8.0:1547251633.082684:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012445984, offset 0 00000400:00000200:8.0:1547251633.082686:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251633.082689:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251633.082692:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04920 00000400:00000200:8.0:1547251633.082694:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd48b9 [1] + 0 00000400:00000200:8.0:1547251633.082697:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d688bb0 00000100:00000200:8.0:1547251633.082698:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251633.082707:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688bb0 00000400:00000200:8.0:1547251633.082708:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251633.082712:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824756600 x1622384012445984/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251640 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251633.082718:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000400:00000200:8.0:1547251633.082719:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251633.082721:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251633.082729:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012445984 00010000:00000200:9.0:1547251633.082739:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3da450 x1622384012445984/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:694/0 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251633.082743:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446000, portal 4 00000100:00000200:9.0:1547251633.082746:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012445984, offset 192 00000100:00000200:6.0:1547251633.082746:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446000, offset 0 00000400:00000200:9.0:1547251633.082748:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251633.082749:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251633.082750:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251633.082752:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251633.082753:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04920 00000400:00000200:9.0:1547251633.082755:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf111 [1] + 192 00000400:00000200:6.0:1547251633.082755:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04930 00000400:00000200:6.0:1547251633.082757:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe26d5 [1] + 0 00000100:00000200:9.0:1547251633.082758:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824756600 x1622384012445984/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251640 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251633.082759:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990f68 00000100:00000200:6.0:1547251633.082761:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251633.082767:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251633.082767:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990f68 00000400:00000200:6.0:1547251633.082768:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251633.082769:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea7f8 00000400:00000200:9.0:1547251633.082770:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251633.082772:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251633.082772:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37936c00 x1622384012446000/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251640 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251633.082776:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824756600 x1622384012445984/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251640 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251633.082779:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:6.0:1547251633.082779:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459903b8 00000400:00000200:6.0:1547251633.082780:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:8.0:1547251633.082782:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824756600 x1622384012445984/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251640 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251633.082782:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:8.0:1547251633.082788:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000100:00000200:7.0:1547251633.082790:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446000 00010000:00080000:8.0:1547251633.082792:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824756600 x1622384012445984/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251640 ref 1 fl Rpc:R/0/0 rc 0/0 00010000:00000200:7.0:1547251633.082800:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39a450 x1622384012446000/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:694/0 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:7.0:1547251633.082808:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446000, offset 192 00000400:00000200:7.0:1547251633.082810:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251633.082813:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251633.082815:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04930 00000400:00000200:7.0:1547251633.082818:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf121 [1] + 192 00000100:00000200:7.0:1547251633.082821:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37936c00 x1622384012446000/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251640 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251633.082829:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251633.082831:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eaee0 00000400:00000200:7.0:1547251633.082832:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251633.082835:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251633.082839:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37936c00 x1622384012446000/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251640 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251633.082841:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251633.082846:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37936c00 x1622384012446000/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251640 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251633.082852:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990330 00010000:00080000:6.0:1547251633.082856:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37936c00 x1622384012446000/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251640 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251633.098683:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446016, portal 4 00000100:00000200:4.0:1547251633.098686:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446016, offset 0 00000400:00000200:4.0:1547251633.098689:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251633.098692:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251633.098694:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04940 00000400:00000200:4.0:1547251633.098697:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2705 [1] + 0 00000400:00000200:4.0:1547251633.098699:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990770 00000100:00000200:4.0:1547251633.098701:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:4.0:1547251633.098709:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990770 00000400:00000200:4.0:1547251633.098710:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251633.098712:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c436900 x1622384012446016/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251640 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251633.098717:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bbb0 00000400:00000200:4.0:1547251633.098718:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251633.098719:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251633.098739:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446016 00010000:00000200:7.0:1547251633.098748:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d39a050 x1622384012446016/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:694/0 lens 224/368 e 0 to 0 dl 1547251639 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:7.0:1547251633.098755:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446016, offset 192 00000400:00000200:7.0:1547251633.098757:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251633.098760:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251633.098763:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04940 00000400:00000200:7.0:1547251633.098766:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf131 [1] + 192 00000100:00000200:7.0:1547251633.098769:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c436900 x1622384012446016/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251640 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251633.098778:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251633.098780:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea000 00000400:00000200:7.0:1547251633.098781:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251633.098782:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251633.098787:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:4.0:1547251633.098787:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c436900 x1622384012446016/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251640 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251633.098791:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c436900 x1622384012446016/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251640 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251633.098795:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b330 00010000:00080000:4.0:1547251633.098798:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae5c436900 x1622384012446016/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251640 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251635.114677:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446032, portal 4 00000100:00000200:11.0:1547251635.114680:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446032, offset 0 00000400:00000200:11.0:1547251635.114683:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:11.0:1547251635.114686:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:11.0:1547251635.114688:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23055] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:11.0:1547251635.114691:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23054] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23055] 00000400:00000200:11.0:1547251635.114693:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:11.0:1547251635.114695:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251635.114696:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251635.114697:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9752, 9754] 00000400:00000200:11.0:1547251635.114700:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:11.0:1547251635.114705:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251635.114707:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:11.0:1547251635.114710:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251635.114712:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:11.0:1547251635.114713:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:11.0:1547251635.114716:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:6.0:1547251635.114718:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446048, portal 4 00000100:00000200:3.0:1547251635.114719:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446064, portal 4 00000100:00000200:6.0:1547251635.114721:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446048, offset 0 00000100:00000200:3.0:1547251635.114722:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446064, offset 0 00000400:00000200:6.0:1547251635.114724:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:3.0:1547251635.114725:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:6.0:1547251635.114727:0:12676:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:3.0:1547251635.114728:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:6.0:1547251635.114730:0:12676:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23055] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251635.114730:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23055] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:6.0:1547251635.114733:0:12676:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23055] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23055] 00000400:00000200:3.0:1547251635.114734:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23055] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23055] 00000800:00000200:9.2:1547251635.114735:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:6.0:1547251635.114736:0:12676:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251635.114736:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:6.0:1547251635.114737:0:12676:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251635.114738:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251635.114739:0:12676:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251635.114739:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251635.114740:0:12676:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9753, 9754] 00000400:00000200:3.0:1547251635.114741:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9753, 9754] 00000400:00000200:6.0:1547251635.114743:0:12676:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:3.0:1547251635.114743:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:6.0:1547251635.114747:0:12676:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:6.0:1547251635.114750:0:12676:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:6.0:1547251635.114753:0:12676:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000400:00000200:3.0:1547251635.114753:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:6.0:1547251635.114754:0:12676:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:6.0:1547251635.114755:0:12676:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:3.0:1547251635.114756:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:6.0:1547251635.114758:0:12676:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:3.0:1547251635.114759:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251635.114761:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (21)++ 00000800:00000200:3.0:1547251635.114762:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (22)++ 00000800:00000200:3.0:1547251635.114765:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (23)-- 00000800:00000200:7.2:1547251635.114777:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (22)++ 00000800:00000200:12.0:1547251635.114792:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251635.114795:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000100:00000200:12.0:1547251635.114799:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07d100 x1622384012446032/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251642 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251635.114804:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b110 00000400:00000200:12.0:1547251635.114805:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251635.114808:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251635.114810:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251635.114812:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (23)++ 00000800:00000200:12.0:1547251635.114813:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (24)-- 00000100:00000200:12.0:1547251635.114816:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37934500 x1622384012446048/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251642 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251635.114820:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459903b8 00000400:00000200:12.0:1547251635.114821:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251635.114823:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251635.114824:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (23)-- 00000800:00000200:12.0:1547251635.114826:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:12.0:1547251635.114827:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:12.0:1547251635.114828:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000100:00000200:12.0:1547251635.114831:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c3300 x1622384012446064/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251642 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251635.114834:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555088 00000800:00000200:14.0:1547251635.114835:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000400:00000200:12.0:1547251635.114835:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251635.114837:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251635.114838:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:7.2:1547251635.114892:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251635.114897:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:14.0:1547251635.114904:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:14.0:1547251635.114908:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251635.114910:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:14.0:1547251635.114912:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251635.114913:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251635.114916:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04960 00000400:00000200:12.0:1547251635.114917:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251635.114920:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fd3d [1] + 192 00000400:00000200:12.0:1547251635.114921:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04950 00000100:00000200:14.0:1547251635.114924:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37934500 x1622384012446048/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251642 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251635.114935:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:12.0:1547251635.114937:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fd2d [1] + 192 00000800:00000200:14.0:1547251635.114939:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000100:00000200:12.0:1547251635.114940:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07d100 x1622384012446032/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251642 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251635.114948:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:14.0:1547251635.114949:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:14.0:1547251635.114950:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:14.0:1547251635.114952:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251635.114955:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251635.114958:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04970 00000400:00000200:12.0:1547251635.114958:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251635.114960:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000100:00000200:6.0:1547251635.114960:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37934500 x1622384012446048/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251642 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251635.114962:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251635.114963:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251635.114964:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:12.0:1547251635.114965:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000100:00000200:6.0:1547251635.114966:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37934500 x1622384012446048/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251642 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251635.114970:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990330 00000400:00000200:14.0:1547251635.114973:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fd4d [1] + 192 00010000:00080000:6.0:1547251635.114974:0:12676:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ae37934500 x1622384012446048/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251642 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251635.114977:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07d100 x1622384012446032/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251642 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251635.114982:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07d100 x1622384012446032/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251642 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251635.114986:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b908 00000100:00000200:14.0:1547251635.114989:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c3300 x1622384012446064/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251642 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:11.0:1547251635.114990:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b17b07d100 x1622384012446032/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251642 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:14.0:1547251635.114997:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251635.114998:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (21)++ 00000800:00000200:14.0:1547251635.114999:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (22)-- 00000800:00000200:14.0:1547251635.115000:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:3.0:1547251635.115006:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c3300 x1622384012446064/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251642 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251635.115011:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c3300 x1622384012446064/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251642 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251635.115016:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251635.115019:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c3300 x1622384012446064/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251642 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:12.0:1547251637.226723:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446080, portal 4 00000100:00000200:12.0:1547251637.226727:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446080, offset 0 00000400:00000200:12.0:1547251637.226731:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:12.0:1547251637.226734:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000100:00000200:9.0:1547251637.226734:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446096, portal 4 00000400:00000200:12.0:1547251637.226736:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23057] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:9.0:1547251637.226736:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446096, offset 0 00000400:00000200:12.0:1547251637.226740:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23055] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23057] 00000400:00000200:9.0:1547251637.226740:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:12.0:1547251637.226742:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251637.226742:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:12.0:1547251637.226743:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251637.226745:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251637.226745:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23057] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251637.226746:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9759, 9759] 00000400:00000200:12.0:1547251637.226748:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:9.0:1547251637.226748:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23056] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23057] 00000400:00000200:9.0:1547251637.226751:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000100:00000200:3.0:1547251637.226751:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446112, portal 4 00000400:00000200:9.0:1547251637.226753:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:3.0:1547251637.226753:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446112, offset 0 00000400:00000200:9.0:1547251637.226754:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251637.226756:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9759, 9760] 00000400:00000200:3.0:1547251637.226756:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251637.226757:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:3.0:1547251637.226759:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251637.226762:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000400:00000200:3.0:1547251637.226762:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23057] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:9.0:1547251637.226765:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:3.0:1547251637.226765:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23057] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23057] 00000400:00000200:12.0:1547251637.226768:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000400:00000200:3.0:1547251637.226768:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251637.226769:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251637.226771:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251637.226772:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9760, 9760] 00000800:00000200:12.0:1547251637.226773:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:9.0:1547251637.226774:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000400:00000200:3.0:1547251637.226774:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000800:00000200:9.0:1547251637.226775:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:12.0:1547251637.226776:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251637.226776:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:12.0:1547251637.226778:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:12.0:1547251637.226779:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:9.0:1547251637.226779:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000400:00000200:3.0:1547251637.226779:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251637.226781:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:12.0:1547251637.226782:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:3.0:1547251637.226784:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251637.226786:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:3.0:1547251637.226787:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:3.0:1547251637.226789:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000800:00000200:7.2:1547251637.226829:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000800:00000200:7.2:1547251637.226838:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:5.2:1547251637.226846:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251637.226849:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251637.226853:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000100:00000200:11.0:1547251637.226858:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf4b00 x1622384012446096/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251637.226861:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251637.226864:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000400:00000200:11.0:1547251637.226865:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea5d8 00000400:00000200:11.0:1547251637.226867:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:10.0:1547251637.226867:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a4200 x1622384012446080/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251637.226871:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251637.226873:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:10.0:1547251637.226874:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b0384c8 00000400:00000200:10.0:1547251637.226875:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251637.226876:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251637.226877:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000400:00000200:10.0:1547251637.226878:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000100:00000200:11.0:1547251637.226880:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c3000 x1622384012446112/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251637.226880:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251637.226882:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:10.0:1547251637.226884:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:10.0:1547251637.226885:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:11.0:1547251637.226886:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555550 00000400:00000200:11.0:1547251637.226887:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251637.226890:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251637.226892:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:7.2:1547251637.226948:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:5.2:1547251637.226953:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:5.2:1547251637.226961:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:10.0:1547251637.226965:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251637.226968:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251637.226968:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251637.226971:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251637.226973:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251637.226976:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251637.226977:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04980 00000400:00000200:11.0:1547251637.226980:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04990 00000400:00000200:10.0:1547251637.226982:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd70b9 [1] + 192 00000400:00000200:11.0:1547251637.226985:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd70c9 [1] + 192 00000100:00000200:10.0:1547251637.226990:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a4200 x1622384012446080/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251637.227002:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251637.227004:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf4b00 x1622384012446096/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251637.227004:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251637.227006:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251637.227007:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251637.227009:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251637.227012:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251637.227015:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:12.0:1547251637.227017:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a4200 x1622384012446080/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251637.227017:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000400:00000200:10.0:1547251637.227017:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251637.227018:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251637.227020:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:10.0:1547251637.227020:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c049a0 00000800:00000200:11.0:1547251637.227022:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:12.0:1547251637.227023:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a4200 x1622384012446080/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251637.227023:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:11.0:1547251637.227024:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:12.0:1547251637.227027:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038dd0 00000400:00000200:10.0:1547251637.227030:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd70e9 [1] + 192 00010000:00080000:12.0:1547251637.227031:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a4200 x1622384012446080/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251637.227035:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf4b00 x1622384012446096/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251637.227041:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf4b00 x1622384012446096/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251637.227045:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea220 00000100:00000200:10.0:1547251637.227048:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c3000 x1622384012446112/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251637.227048:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf4b00 x1622384012446096/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251637.227057:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251637.227058:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251637.227059:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251637.227060:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:3.0:1547251637.227116:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c3000 x1622384012446112/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251637.227122:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c3000 x1622384012446112/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251637.227131:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251637.227135:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c3000 x1622384012446112/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:14.0:1547251637.690692:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446144, portal 4 00000100:00000200:14.0:1547251637.690695:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446144, offset 0 00000100:00000200:9.0:1547251637.690697:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446128, portal 4 00000400:00000200:14.0:1547251637.690699:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000100:00000200:9.0:1547251637.690700:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446128, offset 0 00000400:00000200:14.0:1547251637.690701:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251637.690703:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:14.0:1547251637.690704:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23058] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251637.690706:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:14.0:1547251637.690707:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23057] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23058] 00000400:00000200:9.0:1547251637.690708:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23058] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:14.0:1547251637.690709:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:14.0:1547251637.690711:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251637.690711:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23058] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23058] 00000400:00000200:14.0:1547251637.690712:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251637.690713:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9752, 9753] 00000400:00000200:9.0:1547251637.690713:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:14.0:1547251637.690715:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:9.0:1547251637.690715:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251637.690716:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251637.690718:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9753, 9753] 00000400:00000200:14.0:1547251637.690720:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:9.0:1547251637.690720:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:14.0:1547251637.690723:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:14.0:1547251637.690726:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:9.0:1547251637.690726:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251637.690727:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:14.0:1547251637.690728:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:14.0:1547251637.690731:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:9.0:1547251637.690731:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:9.0:1547251637.690734:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251637.690736:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:9.0:1547251637.690737:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:9.0:1547251637.690739:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:0.2:1547251637.690739:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000100:00000200:3.0:1547251637.690750:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446160, portal 4 00000100:00000200:3.0:1547251637.690754:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446160, offset 0 00000800:00000200:12.2:1547251637.690756:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000400:00000200:3.0:1547251637.690758:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:3.0:1547251637.690761:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:3.0:1547251637.690763:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23059] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251637.690766:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23058] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23059] 00000400:00000200:3.0:1547251637.690768:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:3.0:1547251637.690770:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251637.690771:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251637.690772:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9753, 9754] 00000400:00000200:3.0:1547251637.690774:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:3.0:1547251637.690779:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251637.690782:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:3.0:1547251637.690785:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251637.690786:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:3.0:1547251637.690787:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (23)++ 00000800:00000200:3.0:1547251637.690790:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:5.0:1547251637.690796:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)++ 00000800:00000200:5.0:1547251637.690799:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:4.0:1547251637.690799:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251637.690802:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000100:00000200:5.0:1547251637.690804:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a2100 x1622384012446144/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251637.690805:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf4800 x1622384012446128/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251637.690811:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea000 00000400:00000200:5.0:1547251637.690812:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:5.0:1547251637.690813:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251637.690813:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251637.690817:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251637.690817:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251637.690820:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251637.690822:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251637.690823:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251637.690824:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:4.0:1547251637.690825:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251637.690826:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:4.0:1547251637.690828:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b7200 x1622384012446160/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251637.690834:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555ee0 00000400:00000200:4.0:1547251637.690835:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251637.690838:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251637.690839:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:0.2:1547251637.690904:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251637.690909:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:12.2:1547251637.690915:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:5.0:1547251637.690918:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251637.690923:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251637.690923:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251637.690926:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251637.690928:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251637.690930:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251637.690933:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c049c0 00000400:00000200:4.0:1547251637.690934:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c049b0 00000400:00000200:5.0:1547251637.690937:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3f75 [1] + 192 00000400:00000200:4.0:1547251637.690952:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3f85 [1] + 192 00000100:00000200:5.0:1547251637.690962:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a2100 x1622384012446144/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251637.690975:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251637.690977:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000100:00000200:4.0:1547251637.690977:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf4800 x1622384012446128/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251637.690979:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251637.690981:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251637.690983:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251637.690985:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251637.690988:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:14.0:1547251637.690990:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a2100 x1622384012446144/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251637.690990:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251637.690990:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251637.690991:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251637.690992:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:5.0:1547251637.690994:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c049d0 00000800:00000200:4.0:1547251637.690994:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:14.0:1547251637.690996:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a2100 x1622384012446144/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251637.690996:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:4.0:1547251637.690997:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:14.0:1547251637.691000:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00000100:00000200:9.0:1547251637.691003:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf4800 x1622384012446128/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:14.0:1547251637.691004:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a2100 x1622384012446144/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251637.691009:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf4800 x1622384012446128/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251637.691013:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea220 00000400:00000200:5.0:1547251637.691016:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe3f95 [1] + 192 00010000:00080000:9.0:1547251637.691017:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf4800 x1622384012446128/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251637.691021:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b7200 x1622384012446160/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251637.691030:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251637.691032:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251637.691033:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251637.691034:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000100:00000200:3.0:1547251637.691040:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b7200 x1622384012446160/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251637.691045:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b7200 x1622384012446160/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251637.691049:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251637.691053:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b7200 x1622384012446160/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251644 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251638.090681:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446176, portal 4 00000100:00000200:9.0:1547251638.090684:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446176, offset 0 00000400:00000200:9.0:1547251638.090686:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251638.090689:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251638.090691:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c049e0 00000400:00000200:9.0:1547251638.090694:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4929 [1] + 0 00000400:00000200:9.0:1547251638.090696:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ae6a914a18 00000100:00000200:9.0:1547251638.090698:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251638.090706:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914a18 00000400:00000200:9.0:1547251638.090708:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251638.090712:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf0000 x1622384012446176/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251645 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251638.090718:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea660 00000400:00000200:9.0:1547251638.090719:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251638.090720:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251638.090729:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446176 00010000:00000200:8.0:1547251638.090744:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3da050 x1622384012446176/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:699/0 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251638.090747:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446192, portal 4 00000100:00000200:8.0:1547251638.090750:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446176, offset 192 00000100:00000200:5.0:1547251638.090751:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446192, offset 0 00000400:00000200:8.0:1547251638.090752:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251638.090754:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251638.090755:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251638.090757:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c049e0 00000400:00000200:5.0:1547251638.090757:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251638.090760:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf171 [1] + 192 00000400:00000200:5.0:1547251638.090760:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c049f0 00000100:00000200:8.0:1547251638.090763:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf0000 x1622384012446176/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251645 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251638.090763:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2775 [1] + 0 00000400:00000200:5.0:1547251638.090765:0:12675:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17eadd0 00000100:00000200:5.0:1547251638.090767:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251638.090771:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251638.090773:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688198 00000400:00000200:8.0:1547251638.090774:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251638.090776:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eadd0 00000400:00000200:8.0:1547251638.090777:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:5.0:1547251638.090777:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251638.090781:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf0000 x1622384012446176/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251645 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251638.090781:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df5a00 x1622384012446192/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251645 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251638.090783:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:9.0:1547251638.090787:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf0000 x1622384012446176/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251645 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251638.090788:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad5d8 00000400:00000200:5.0:1547251638.090789:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251638.090790:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:9.0:1547251638.090793:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea220 00010000:00080000:9.0:1547251638.090798:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf0000 x1622384012446176/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251645 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251638.090802:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446192 00010000:00000200:4.0:1547251638.090812:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d399c50 x1622384012446192/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:699/0 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251638.090818:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446192, offset 192 00000400:00000200:4.0:1547251638.090820:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251638.090823:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251638.090825:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c049f0 00000400:00000200:4.0:1547251638.090828:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf181 [1] + 192 00000100:00000200:4.0:1547251638.090831:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df5a00 x1622384012446192/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251645 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251638.090840:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251638.090843:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b330 00000400:00000200:4.0:1547251638.090845:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251638.090847:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251638.090851:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df5a00 x1622384012446192/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251645 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251638.090853:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251638.090857:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df5a00 x1622384012446192/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251645 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251638.090862:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad2a8 00010000:00080000:5.0:1547251638.090866:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df5a00 x1622384012446192/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251645 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251638.106702:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446208, portal 4 00000100:00000200:5.0:1547251638.106705:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446208, offset 0 00000400:00000200:5.0:1547251638.106708:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251638.106711:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251638.106713:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04a00 00000400:00000200:5.0:1547251638.106716:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe27a5 [1] + 0 00000400:00000200:5.0:1547251638.106718:0:12675:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17ea880 00000100:00000200:5.0:1547251638.106720:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:5.0:1547251638.106728:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea880 00000400:00000200:5.0:1547251638.106729:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251638.106733:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b82444ad00 x1622384012446208/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251645 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251638.106741:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad5d8 00000400:00000200:5.0:1547251638.106742:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251638.106743:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251638.106751:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446208 00010000:00000200:4.0:1547251638.106761:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d399850 x1622384012446208/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:699/0 lens 224/368 e 0 to 0 dl 1547251644 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251638.106768:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446208, offset 192 00000400:00000200:4.0:1547251638.106770:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251638.106773:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251638.106775:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04a00 00000400:00000200:4.0:1547251638.106778:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf191 [1] + 192 00000100:00000200:4.0:1547251638.106780:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b82444ad00 x1622384012446208/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251645 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251638.106789:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251638.106791:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bbb0 00000400:00000200:4.0:1547251638.106792:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251638.106795:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251638.106799:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b82444ad00 x1622384012446208/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251645 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251638.106801:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251638.106805:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b82444ad00 x1622384012446208/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251645 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251638.106810:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad2a8 00010000:00080000:5.0:1547251638.106814:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b82444ad00 x1622384012446208/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251645 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:0.0:1547251640.122690:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446224, portal 4 00000100:00000200:0.0:1547251640.122693:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446224, offset 0 00000400:00000200:0.0:1547251640.122696:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:0.0:1547251640.122699:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:0.0:1547251640.122701:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23059] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251640.122704:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23059] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23059] 00000400:00000200:0.0:1547251640.122706:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:0.0:1547251640.122709:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251640.122710:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251640.122712:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9753, 9756] 00000400:00000200:0.0:1547251640.122713:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:0.0:1547251640.122718:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251640.122721:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:0.0:1547251640.122724:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251640.122725:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:0.0:1547251640.122726:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:0.0:1547251640.122729:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:4.0:1547251640.122743:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446240, portal 4 00000100:00000200:4.0:1547251640.122747:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446240, offset 0 00000800:00000200:7.2:1547251640.122749:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:4.0:1547251640.122752:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:4.0:1547251640.122755:0:12676:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:4.0:1547251640.122757:0:12676:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23060] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:4.0:1547251640.122760:0:12676:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23059] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23060] 00000400:00000200:4.0:1547251640.122763:0:12676:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:4.0:1547251640.122764:0:12676:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251640.122766:0:12676:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251640.122767:0:12676:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9754, 9756] 00000400:00000200:4.0:1547251640.122769:0:12676:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:4.0:1547251640.122773:0:12676:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:4.0:1547251640.122776:0:12676:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:4.0:1547251640.122781:0:12676:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:4.0:1547251640.122782:0:12676:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000100:00000200:10.0:1547251640.122783:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446256, portal 4 00000800:00000200:4.0:1547251640.122783:0:12676:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000100:00000200:10.0:1547251640.122785:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446256, offset 0 00000800:00000200:4.0:1547251640.122786:0:12676:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:10.0:1547251640.122789:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:10.0:1547251640.122791:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:10.0:1547251640.122794:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23060] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:10.0:1547251640.122797:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23060] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23060] 00000400:00000200:10.0:1547251640.122799:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:9.2:1547251640.122799:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:10.0:1547251640.122800:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251640.122802:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251640.122803:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9754, 9757] 00000400:00000200:10.0:1547251640.122805:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:10.0:1547251640.122810:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:10.0:1547251640.122812:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:10.0:1547251640.122815:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:10.0:1547251640.122817:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251640.122818:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:10.0:1547251640.122818:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (23)++ 00000800:00000200:12.0:1547251640.122821:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (25)-- 00000800:00000200:10.0:1547251640.122821:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (25)-- 00000100:00000200:12.0:1547251640.122825:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f5700 x1622384012446224/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251647 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251640.122830:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665dd0 00000400:00000200:12.0:1547251640.122831:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251640.122834:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251640.122836:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251640.122838:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:12.0:1547251640.122839:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:12.0:1547251640.122842:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df2400 x1622384012446240/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251647 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251640.122846:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b088 00000400:00000200:12.0:1547251640.122847:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251640.122849:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251640.122850:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251640.122851:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251640.122852:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:12.0:1547251640.122854:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960ecb00 x1622384012446256/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251647 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251640.122855:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:14.0:1547251640.122857:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:12.0:1547251640.122862:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914aa0 00000400:00000200:12.0:1547251640.122863:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251640.122865:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251640.122866:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:7.2:1547251640.122869:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:14.0:1547251640.122883:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:14.0:1547251640.122892:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.1.11.93@o2ib10 00000800:00000200:14.0:1547251640.122894:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:14.0:1547251640.122896:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:14.0:1547251640.122896:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:14.0:1547251640.122898:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:9.2:1547251640.122907:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:7.2:1547251640.122914:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:12.0:1547251640.122919:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251640.122922:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000800:00000200:14.0:1547251640.122925:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:12.0:1547251640.122925:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:14.0:1547251640.122927:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000800:00000200:9.2:1547251640.122927:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000400:00000200:12.0:1547251640.122929:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04a10 00000400:00000200:14.0:1547251640.122931:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251640.122932:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fd7d [1] + 192 00000400:00000200:14.0:1547251640.122935:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04a30 00000100:00000200:12.0:1547251640.122936:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f5700 x1622384012446224/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251647 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251640.122954:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:14.0:1547251640.122956:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fd9d [1] + 192 00000800:00000200:12.0:1547251640.122956:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251640.122958:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251640.122959:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:14.0:1547251640.122960:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960ecb00 x1622384012446256/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251647 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251640.122961:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:12.0:1547251640.122963:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251640.122964:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251640.122966:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000400:00000200:14.0:1547251640.122969:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:12.0:1547251640.122969:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:14.0:1547251640.122971:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:0.0:1547251640.122972:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f5700 x1622384012446224/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251647 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:14.0:1547251640.122973:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:12.0:1547251640.122973:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04a20 00000800:00000200:14.0:1547251640.122974:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:14.0:1547251640.122975:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:0.0:1547251640.122977:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f5700 x1622384012446224/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251647 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251640.122982:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00000400:00000200:12.0:1547251640.122984:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fd8d [1] + 192 00010000:00080000:0.0:1547251640.122986:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f5700 x1622384012446224/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251647 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251640.122988:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960ecb00 x1622384012446256/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251647 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:10.0:1547251640.122994:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960ecb00 x1622384012446256/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251647 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251640.123006:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914dd0 00000100:00000200:12.0:1547251640.123008:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df2400 x1622384012446240/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251647 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:10.0:1547251640.123010:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba960ecb00 x1622384012446256/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251647 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:12.0:1547251640.123017:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251640.123018:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251640.123020:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251640.123021:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:4.0:1547251640.123027:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df2400 x1622384012446240/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251647 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251640.123032:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df2400 x1622384012446240/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251647 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251640.123037:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bc38 00010000:00080000:4.0:1547251640.123040:0:12676:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a765df2400 x1622384012446240/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251647 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251642.234683:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446272, portal 4 00000100:00000200:8.0:1547251642.234686:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446272, offset 0 00000400:00000200:8.0:1547251642.234689:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251642.234691:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251642.234694:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23061] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251642.234697:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23060] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23061] 00000400:00000200:8.0:1547251642.234699:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:8.0:1547251642.234700:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251642.234702:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251642.234703:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9760, 9761] 00000400:00000200:8.0:1547251642.234705:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:8.0:1547251642.234710:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251642.234712:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:8.0:1547251642.234715:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251642.234716:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:8.0:1547251642.234717:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:8.0:1547251642.234720:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000100:00000200:14.0:1547251642.234739:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446288, portal 4 00000100:00000200:2.0:1547251642.234740:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446304, portal 4 00000100:00000200:14.0:1547251642.234743:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446288, offset 0 00000100:00000200:2.0:1547251642.234743:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446304, offset 0 00000400:00000200:2.0:1547251642.234746:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:14.0:1547251642.234748:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:2.0:1547251642.234749:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:14.0:1547251642.234751:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:2.0:1547251642.234752:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23061] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:14.0:1547251642.234753:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23061] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251642.234755:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23061] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23061] 00000400:00000200:2.0:1547251642.234757:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:14.0:1547251642.234759:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23061] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23061] 00000400:00000200:2.0:1547251642.234759:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251642.234760:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251642.234761:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:2.0:1547251642.234762:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9761, 9761] 00000400:00000200:14.0:1547251642.234763:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251642.234763:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:14.0:1547251642.234764:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251642.234765:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9761, 9762] 00000400:00000200:14.0:1547251642.234768:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:2.0:1547251642.234768:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251642.234771:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:14.0:1547251642.234774:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:7.2:1547251642.234774:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000800:00000200:2.0:1547251642.234774:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251642.234775:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:14.0:1547251642.234777:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:2.0:1547251642.234777:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:2.0:1547251642.234779:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000800:00000200:14.0:1547251642.234780:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251642.234782:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (21)++ 00000800:00000200:14.0:1547251642.234783:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (22)++ 00000800:00000200:14.0:1547251642.234785:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251642.234798:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251642.234806:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000100:00000200:11.0:1547251642.234812:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824752a00 x1622384012446272/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251642.234814:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:11.0:1547251642.234819:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688bb0 00000400:00000200:11.0:1547251642.234821:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251642.234825:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251642.234827:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:5.2:1547251642.234831:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251642.234847:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (23)++ 00000800:00000200:10.0:1547251642.234853:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (24)-- 00000100:00000200:10.0:1547251642.234858:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f5400 x1622384012446304/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251642.234859:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (23)++ 00000800:00000200:11.0:1547251642.234861:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (24)-- 00000100:00000200:11.0:1547251642.234864:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af4500 x1622384012446288/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251642.234865:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06aa18 00000400:00000200:10.0:1547251642.234866:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251642.234869:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:10.0:1547251642.234869:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251642.234870:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251642.234872:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (23)-- 00000400:00000200:11.0:1547251642.234873:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251642.234874:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251642.234875:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:7.2:1547251642.234907:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:5.2:1547251642.234913:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251642.234919:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:10.0:1547251642.234921:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251642.234926:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251642.234926:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251642.234928:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251642.234931:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251642.234933:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251642.234936:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04a40 00000400:00000200:11.0:1547251642.234937:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04a60 00000400:00000200:10.0:1547251642.234940:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7149 [1] + 192 00000400:00000200:11.0:1547251642.234952:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7179 [1] + 192 00000100:00000200:10.0:1547251642.234957:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824752a00 x1622384012446272/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251642.234968:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251642.234969:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f5400 x1622384012446304/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251642.234970:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251642.234972:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251642.234973:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251642.234975:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251642.234978:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251642.234979:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251642.234981:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000400:00000200:10.0:1547251642.234982:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000100:00000200:8.0:1547251642.234982:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824752a00 x1622384012446272/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251642.234983:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251642.234985:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251642.234985:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04a50 00000800:00000200:11.0:1547251642.234987:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:8.0:1547251642.234987:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824752a00 x1622384012446272/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251642.234988:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:11.0:1547251642.234989:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:8.0:1547251642.234991:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00010000:00080000:8.0:1547251642.234995:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824752a00 x1622384012446272/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:2.0:1547251642.234995:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f5400 x1622384012446304/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251642.235000:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f5400 x1622384012446304/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251642.235005:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06ab28 00000400:00000200:10.0:1547251642.235009:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7169 [1] + 192 00010000:00080000:2.0:1547251642.235009:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f5400 x1622384012446304/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251642.235013:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af4500 x1622384012446288/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251642.235024:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251642.235025:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251642.235026:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251642.235027:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:14.0:1547251642.235034:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af4500 x1622384012446288/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:14.0:1547251642.235039:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af4500 x1622384012446288/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251642.235043:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00010000:00080000:14.0:1547251642.235047:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af4500 x1622384012446288/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:0.2:1547251642.602841:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b81d647a00] (20)++ 00000800:00000200:14.0:1547251642.602898:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:14.0:1547251642.602909:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.10.184@o2ib10 00000400:00000200:14.0:1547251642.602913:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.10.184@o2ib10 : PUT - for me 00000400:00000200:14.0:1547251642.602917:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.10.184@o2ib10 of length 48 into portal 0 MB=0x8000000000000000 00000400:00000200:14.0:1547251642.602920:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 0 from 12345-10.0.10.184@o2ib10 of length 48/48 into md 0x11 [1] + 0 00000400:00000200:14.0:1547251642.602925:0:12666:0:(peer.c:1955:lnet_peer_push_event()) Stale Push from 10.0.10.184@o2ib10: got 1 have 1 00000400:00000200:14.0:1547251642.602926:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:14.0:1547251642.602929:0:12666:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.10.184@o2ib10 local destination 00000400:00000200:14.0:1547251642.602931:0:12666:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251642.602933:0:12666:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251642.602934:0:12666:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.0.10.184@o2ib10 c:[8, 8], s:[16, 17] 00000400:00000200:14.0:1547251642.602937:0:12666:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.10.184@o2ib10 00000400:00000200:14.0:1547251642.602950:0:12666:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.10.184@o2ib10(10.0.10.184@o2ib10:10.0.10.184@o2ib10) : ACK try# 0 00000800:00000200:14.0:1547251642.602953:0:12666:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 0 bytes in 1 frags to 12345-10.0.10.184@o2ib10 00000800:00000200:14.0:1547251642.602956:0:12666:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b971916c00] -> 10.0.10.184@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251642.602957:0:12666:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b81d647a00] (22)++ 00000800:00000200:14.0:1547251642.602958:0:12666:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b81d647a00] (23)++ 00000800:00000200:12.0:1547251642.602960:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (24)-- 00000800:00000200:14.0:1547251642.602961:0:12666:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b81d647a00] (24)-- 00000800:00000200:14.0:1547251642.602962:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (22)++ 00000800:00000200:14.0:1547251642.602963:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (23)-- 00000800:00000200:14.0:1547251642.602964:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)-- 00000800:00000200:0.2:1547251642.602974:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:12.0:1547251642.602987:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)++ 00000800:00000200:12.0:1547251642.602996:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b81d647a00] (23)-- 00000400:00000200:12.0:1547251642.602998:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251642.603001:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.10.184@o2ib10: ACK: OK 00000800:00000200:14.0:1547251642.603003:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)-- 00000800:00000200:12.0:1547251642.603004:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (21)-- 00000800:00000200:8.2:1547251642.603063:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d02b200] (20)++ 00000800:00000200:14.0:1547251642.603077:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b200] (21)++ 00000800:00000200:14.0:1547251642.603083:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.10.184@o2ib10 00000400:00000200:14.0:1547251642.603086:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.10.184@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251642.603089:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b200] (22)-- 00000400:00000200:14.0:1547251642.603090:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.10.184@o2ib10 of length 520 into portal 26 MB=0x5c3934d100010 00000400:00000200:14.0:1547251642.603092:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.10.184@o2ib10 of length 520/520 into md 0xd5279 [1] + 0 00000400:00000200:14.0:1547251642.603095:0:12666:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d6882a8 00000100:00000200:14.0:1547251642.603096:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:14.0:1547251642.603101:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6882a8 00000400:00000200:14.0:1547251642.603102:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:14.0:1547251642.603104:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (21)++ 00000800:00000200:14.0:1547251642.603106:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (22)-- 00000800:00000200:14.0:1547251642.603107:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b200] (21)-- 00000100:00000200:2.0:1547251642.603159:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622412304056336 00010000:00080000:2.0:1547251642.603170:0:12929:0:(ldlm_lib.c:1228:target_handle_connect()) MGS: connection from 8534aa65-7ac8-198c-0718-3e6f83d57705@10.0.10.184@o2ib10 t0 exp (null) cur 1547251642 last 0 00000020:00000080:2.0:1547251642.603181:0:12929:0:(genops.c:1419:class_connect()) connect: client 8534aa65-7ac8-198c-0718-3e6f83d57705, cookie 0xdec335f6b872ab88 00000020:01000000:2.0:1547251642.603184:0:12929:0:(lprocfs_status_server.c:467:lprocfs_exp_setup()) using hash ffff97b9e8704300 00000400:00000200:2.0:1547251642.603192:0:12929:0:(peer.c:1106:LNetPrimaryNID()) NID 10.0.10.184@o2ib10 primary NID 10.0.10.184@o2ib10 rc 0 00000100:00080000:2.0:1547251642.603196:0:12929:0:(import.c:104:ptlrpc_import_enter_resend()) ffff97b824577000 : changing import state from NEW to RECOVER 00000100:00080000:2.0:1547251642.603200:0:12929:0:(import.c:1528:ptlrpc_import_recovery_state_machine()) ffff97b824577000 : changing import state from RECOVER to FULL 00000100:02000000:2.0:1547251642.603202:0:12929:0:(import.c:1534:ptlrpc_import_recovery_state_machine()) MGS: Connection restored to f7adbd22-dd00-137c-616b-9c8e12a50a01 (at 10.0.10.184@o2ib10) 00010000:00000200:2.0:1547251642.603213:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692fec850 x1622412304056336/t0(0) o250->8534aa65-7ac8-198c-0718-3e6f83d57705@10.0.10.184@o2ib10:0/0 lens 520/416 e 0 to 0 dl 1547251742 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:2.0:1547251642.603219:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 25, xid 1622412304056336, offset 0 00000400:00000200:2.0:1547251642.603222:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.10.184@o2ib10 00000400:00000200:2.0:1547251642.603225:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.10.184@o2ib10 local destination 00000400:00000200:2.0:1547251642.603227:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251642.603229:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251642.603230:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.0.10.184@o2ib10 c:[8, 8], s:[17, 17] 00000400:00000200:2.0:1547251642.603232:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.2.10.184@o2ib10 00000400:00000200:2.0:1547251642.603237:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.2.10.184@o2ib10(10.0.10.184@o2ib10:10.2.10.184@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251642.603240:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 384 bytes in 1 frags to 12345-10.2.10.184@o2ib10 00000800:00000200:2.0:1547251642.603243:0:12929:0:(o2iblnd_cb.c:1604:kiblnd_launch_tx()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (1)++ 00000800:00000200:2.0:1547251642.603246:0:12929:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:2.0:1547251642.603262:0:12929:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:2.0:1547251642.603264:0:12929:0:(o2iblnd_cb.c:1611:kiblnd_launch_tx()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000100:00000200:2.0:1547251642.603268:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:10.0:1547251642.603309:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251642.603320:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251642.603810:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251642.603817:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b81d44f000 0x13dd44f000(0x13dd44f000) 00000800:00000200:10.0:1547251642.603819:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba92af3000 0x1652af3000(0x1652af3000) 00000800:00000200:10.0:1547251642.603820:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97bae04f3000 0x16a04f3000(0x16a04f3000) 00000800:00000200:10.0:1547251642.603821:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ba9cf29000 0x165cf29000(0x165cf29000) 00000800:00000200:10.0:1547251642.603822:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97bae2635000 0x16a2635000(0x16a2635000) 00000800:00000200:10.0:1547251642.603823:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b824664000 0x13e4664000(0x13e4664000) 00000800:00000200:10.0:1547251642.603824:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b82470c000 0x13e470c000(0x13e470c000) 00000800:00000200:10.0:1547251642.603825:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b81d497000 0x13dd497000(0x13dd497000) 00000800:00000200:10.0:1547251642.603825:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97babd0f1000 0x167d0f1000(0x167d0f1000) 00000800:00000200:10.0:1547251642.603826:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97bae01df000 0x16a01df000(0x16a01df000) 00000800:00000200:10.0:1547251642.603827:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97a99520e000 0x55520e000(0x55520e000) 00000800:00000200:10.0:1547251642.603828:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba9f360000 0x165f360000(0x165f360000) 00000800:00000200:10.0:1547251642.603829:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b9073ab000 0x14c73ab000(0x14c73ab000) 00000800:00000200:10.0:1547251642.603830:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97bae4672000 0x16a4672000(0x16a4672000) 00000800:00000200:10.0:1547251642.603831:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba96314000 0x1656314000(0x1656314000) 00000800:00000200:10.0:1547251642.603832:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba9bae9000 0x165bae9000(0x165bae9000) 00000800:00000200:10.0:1547251642.603833:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b6268c2000 0x11e68c2000(0x11e68c2000) 00000800:00000200:10.0:1547251642.603834:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97badf0ab000 0x169f0ab000(0x169f0ab000) 00000800:00000200:9.0:1547251642.604482:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604484:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604485:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604485:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604486:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604487:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604487:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604488:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604489:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604489:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604490:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604490:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604491:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604492:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604492:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604493:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604493:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604494:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604495:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604495:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604496:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604496:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604497:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604498:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604498:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604499:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604499:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604500:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604501:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604501:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604502:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604502:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604503:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604504:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.604504:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.604506:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000100:12.0:1547251642.604736:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251642.604739:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251642.604741:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251642.604778:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:12.0:1547251642.604784:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:12.0:1547251642.604788:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:12.0:1547251642.604792:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:12.0:1547251642.604793:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (19)-- 00000800:00000200:12.0:1547251642.604794:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (18)-- 00000800:00000200:8.2:1547251642.604920:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c800] (17)++ 00000800:00000200:12.0:1547251642.604931:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (18)++ 00000800:00000200:12.0:1547251642.604935:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (19)-- 00000800:00000200:12.0:1547251642.604936:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (18)-- 00000800:00000200:12.0:1547251642.604937:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (17)++ 00000800:00000200:12.0:1547251642.604938:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (18)-- 00000800:00000200:12.0:1547251642.604938:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (17)-- 00000800:00000200:12.0:1547251642.604939:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (16)++ 00000800:00000200:12.0:1547251642.604940:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (17)-- 00000800:00000200:12.0:1547251642.604941:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (16)-- 00000800:00000200:12.0:1547251642.604946:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (15)++ 00000800:00000200:12.0:1547251642.604947:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (16)-- 00000800:00000200:12.0:1547251642.604948:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (15)-- 00000800:00000200:14.0:1547251642.604950:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (14)++ 00000800:00000200:14.0:1547251642.604956:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (15)-- 00000800:00000200:14.0:1547251642.604957:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (14)-- 00000800:00000200:14.0:1547251642.604959:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (13)++ 00000800:00000200:14.0:1547251642.604961:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (14)-- 00000800:00000200:14.0:1547251642.604962:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (13)-- 00000800:00000200:14.0:1547251642.604963:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (12)++ 00000800:00000200:14.0:1547251642.604964:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (13)-- 00000800:00000200:12.0:1547251642.604965:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (13)++ 00000800:00000200:14.0:1547251642.604966:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (13)-- 00000800:00000200:14.0:1547251642.604967:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (12)++ 00000800:00000200:12.0:1547251642.604967:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (12)-- 00000800:00000200:12.0:1547251642.604968:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (12)-- 00000800:00000200:14.0:1547251642.604969:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (11)-- 00000800:00000200:12.0:1547251642.604969:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (11)++ 00000800:00000200:12.0:1547251642.604970:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (12)-- 00000800:00000200:14.0:1547251642.604971:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (11)-- 00000800:00000200:12.0:1547251642.604972:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (9)-- 00000800:00000200:14.0:1547251642.604973:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (8)++ 00000800:00000200:14.0:1547251642.604976:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (9)-- 00000800:00000200:14.0:1547251642.604977:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (8)-- 00000800:00000200:14.0:1547251642.604978:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (7)++ 00000800:00000200:14.0:1547251642.604979:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (8)-- 00000800:00000200:12.0:1547251642.604979:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (8)++ 00000800:00000200:14.0:1547251642.604980:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (8)-- 00000800:00000200:14.0:1547251642.604981:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (7)++ 00000800:00000200:12.0:1547251642.604981:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (7)-- 00000800:00000200:12.0:1547251642.604982:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (7)-- 00000800:00000200:14.0:1547251642.604983:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (6)-- 00000800:00000200:12.0:1547251642.604983:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (6)++ 00000800:00000200:14.0:1547251642.604984:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (6)-- 00000800:00000200:14.0:1547251642.604985:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (5)++ 00000800:00000200:12.0:1547251642.604985:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (5)-- 00000800:00000200:14.0:1547251642.604986:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (6)-- 00000800:00000200:14.0:1547251642.604987:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (4)-- 00000800:00000200:12.0:1547251642.604987:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (3)-- 00000800:00000200:14.0:1547251642.604988:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (2)++ 00000800:00000200:14.0:1547251642.604991:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (3)-- 00000800:00000200:14.0:1547251642.604992:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (2)-- 00000800:00000200:14.0:1547251642.604993:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (1)-- 00000800:00000200:9.0:1547251642.605007:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.605881:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251642.605886:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.605897:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251642.605899:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251642.605905:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251642.605914:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251642.606419:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251642.606430:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba9603f000 0x165603f000(0x165603f000) 00000800:00000200:10.0:1547251642.606432:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97baa0ff1000 0x1660ff1000(0x1660ff1000) 00000800:00000200:10.0:1547251642.606433:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97af1056b000 0xad056b000(0xad056b000) 00000800:00000200:10.0:1547251642.606434:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97acce267000 0x88e267000(0x88e267000) 00000800:00000200:10.0:1547251642.606435:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97baa2b3c000 0x1662b3c000(0x1662b3c000) 00000800:00000200:10.0:1547251642.606436:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba96eb7000 0x1656eb7000(0x1656eb7000) 00000800:00000200:10.0:1547251642.606437:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b1778dd000 0xd378dd000(0xd378dd000) 00000800:00000200:10.0:1547251642.606438:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba9b8e7000 0x165b8e7000(0x165b8e7000) 00000800:00000200:10.0:1547251642.606439:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97a744b1d000 0x304b1d000(0x304b1d000) 00000800:00000200:10.0:1547251642.606440:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba9e746000 0x165e746000(0x165e746000) 00000800:00000200:10.0:1547251642.606441:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b06f303000 0xc2f303000(0xc2f303000) 00000800:00000200:10.0:1547251642.606442:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97a601bae000 0x1c1bae000(0x1c1bae000) 00000800:00000200:10.0:1547251642.606443:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b34919f000 0xf0919f000(0xf0919f000) 00000800:00000200:10.0:1547251642.606444:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba9dbd2000 0x165dbd2000(0x165dbd2000) 00000800:00000200:10.0:1547251642.606444:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ae3babc000 0x9fbabc000(0x9fbabc000) 00000800:00000200:10.0:1547251642.606445:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b073a9d000 0xc33a9d000(0xc33a9d000) 00000800:00000200:10.0:1547251642.606446:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b17b399000 0xd3b399000(0xd3b399000) 00000800:00000200:10.0:1547251642.606447:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b6fde3e000 0x12bde3e000(0x12bde3e000) 00000800:00000200:9.0:1547251642.607039:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607041:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607041:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607042:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607043:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607043:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607044:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607045:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607045:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607046:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607047:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607047:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607048:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607048:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607049:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607050:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607050:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607051:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607051:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607052:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607053:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607053:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607054:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607054:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607055:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607056:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607056:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607057:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607057:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607058:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607059:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607060:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607060:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607061:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.607062:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.607062:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000100:12.0:1547251642.607293:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251642.607296:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251642.607298:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251642.607337:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:12.0:1547251642.607346:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (20)++ 00000800:00000200:12.0:1547251642.607352:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (21)-- 00000800:00000200:12.0:1547251642.607353:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:12.0:1547251642.607354:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (19)-- 00000800:00000200:12.0:1547251642.607383:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:8.2:1547251642.607481:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c600] (17)++ 00000800:00000200:12.0:1547251642.607492:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)++ 00000800:00000200:12.0:1547251642.607497:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (19)-- 00000800:00000200:12.0:1547251642.607498:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:12.0:1547251642.607498:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (17)++ 00000800:00000200:12.0:1547251642.607499:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:12.0:1547251642.607500:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (17)-- 00000800:00000200:12.0:1547251642.607501:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (16)++ 00000800:00000200:12.0:1547251642.607502:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (17)-- 00000800:00000200:12.0:1547251642.607503:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (16)-- 00000800:00000200:12.0:1547251642.607503:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)++ 00000800:00000200:12.0:1547251642.607507:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (16)-- 00000800:00000200:12.0:1547251642.607507:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)-- 00000800:00000200:12.0:1547251642.607508:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (14)++ 00000800:00000200:12.0:1547251642.607509:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (15)-- 00000800:00000200:12.0:1547251642.607510:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (14)-- 00000800:00000200:15.0:1547251642.607511:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)++ 00000800:00000200:15.0:1547251642.607518:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (14)-- 00000800:00000200:15.0:1547251642.607519:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:15.0:1547251642.607520:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)++ 00000800:00000200:15.0:1547251642.607523:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:15.0:1547251642.607524:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)-- 00000800:00000200:15.0:1547251642.607525:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)++ 00000800:00000200:15.0:1547251642.607528:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (12)-- 00000800:00000200:15.0:1547251642.607529:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)-- 00000800:00000200:15.0:1547251642.607529:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (10)++ 00000800:00000200:15.0:1547251642.607531:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (11)-- 00000800:00000200:12.0:1547251642.607531:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)++ 00000800:00000200:15.0:1547251642.607532:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)-- 00000800:00000200:15.0:1547251642.607533:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (10)++ 00000800:00000200:12.0:1547251642.607533:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (10)-- 00000800:00000200:12.0:1547251642.607534:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (10)-- 00000800:00000200:15.0:1547251642.607535:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (9)-- 00000800:00000200:12.0:1547251642.607535:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (9)++ 00000800:00000200:15.0:1547251642.607536:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (9)-- 00000800:00000200:15.0:1547251642.607537:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)++ 00000800:00000200:12.0:1547251642.607537:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:12.0:1547251642.607538:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:15.0:1547251642.607539:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (7)-- 00000800:00000200:12.0:1547251642.607539:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (7)++ 00000800:00000200:12.0:1547251642.607540:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:12.0:1547251642.607540:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (7)-- 00000800:00000200:12.0:1547251642.607541:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (6)++ 00000800:00000200:12.0:1547251642.607542:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (7)-- 00000800:00000200:12.0:1547251642.607542:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (6)-- 00000800:00000200:12.0:1547251642.607544:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)++ 00000800:00000200:15.0:1547251642.607545:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)-- 00000800:00000200:12.0:1547251642.607545:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (5)-- 00000800:00000200:12.0:1547251642.607546:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (3)-- 00000800:00000200:15.0:1547251642.607547:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (2)++ 00000800:00000200:15.0:1547251642.607551:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (3)-- 00000800:00000200:15.0:1547251642.607551:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (2)-- 00000800:00000200:15.0:1547251642.607553:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (1)-- 00000800:00000200:9.0:1547251642.607565:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.608485:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251642.608490:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.608499:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251642.608500:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:14.0:1547251642.608507:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:14.0:1547251642.608516:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:14.0:1547251642.609048:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:14.0:1547251642.609058:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b8245eb000 0x13e45eb000(0x13e45eb000) 00000800:00000200:14.0:1547251642.609060:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97bae17eb000 0x16a17eb000(0x16a17eb000) 00000800:00000200:14.0:1547251642.609061:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97a954a67000 0x514a67000(0x514a67000) 00000800:00000200:14.0:1547251642.609062:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b907293000 0x14c7293000(0x14c7293000) 00000800:00000200:14.0:1547251642.609062:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ba8f26a000 0x164f26a000(0x164f26a000) 00000800:00000200:14.0:1547251642.609063:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b81d787000 0x13dd787000(0x13dd787000) 00000800:00000200:14.0:1547251642.609064:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ba9e2da000 0x165e2da000(0x165e2da000) 00000800:00000200:14.0:1547251642.609065:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b81d5ad000 0x13dd5ad000(0x13dd5ad000) 00000800:00000200:14.0:1547251642.609066:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97a5c2ccd000 0x182ccd000(0x182ccd000) 00000800:00000200:14.0:1547251642.609067:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ae6a85e000 0xa2a85e000(0xa2a85e000) 00000800:00000200:14.0:1547251642.609068:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97bae2735000 0x16a2735000(0x16a2735000) 00000800:00000200:14.0:1547251642.609068:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b82461b000 0x13e461b000(0x13e461b000) 00000800:00000200:14.0:1547251642.609069:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba974d1000 0x16574d1000(0x16574d1000) 00000800:00000200:14.0:1547251642.609070:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba941ef000 0x16541ef000(0x16541ef000) 00000800:00000200:14.0:1547251642.609071:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b8244d9000 0x13e44d9000(0x13e44d9000) 00000800:00000200:14.0:1547251642.609072:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97badc8ee000 0x169c8ee000(0x169c8ee000) 00000800:00000200:14.0:1547251642.609073:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b8245cf000 0x13e45cf000(0x13e45cf000) 00000800:00000200:14.0:1547251642.609074:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba9c23e000 0x165c23e000(0x165c23e000) 00000800:00000200:9.0:1547251642.609736:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609738:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609738:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609739:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609739:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609740:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609741:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609741:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609742:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609743:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609743:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609744:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609744:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609745:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609746:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609746:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609747:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609748:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609748:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609749:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609749:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609750:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609751:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609751:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609752:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609752:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609753:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609754:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609755:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609755:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609756:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609756:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609757:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609758:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251642.609758:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251642.609759:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000100:12.0:1547251642.609994:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251642.609996:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251642.609998:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251642.610035:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:12.0:1547251642.610042:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (20)++ 00000800:00000200:12.0:1547251642.610047:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (21)-- 00000800:00000200:12.0:1547251642.610047:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:12.0:1547251642.610048:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (19)-- 00000800:00000200:12.0:1547251642.610051:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97babe8ae000] (18)-- 00000800:00000200:8.2:1547251642.610178:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ae000] (17)++ 00000800:00000200:12.0:1547251642.610188:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (18)++ 00000800:00000200:12.0:1547251642.610193:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (19)-- 00000800:00000200:12.0:1547251642.610193:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (18)-- 00000800:00000200:12.0:1547251642.610194:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (17)++ 00000800:00000200:12.0:1547251642.610195:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (18)-- 00000800:00000200:12.0:1547251642.610196:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (17)-- 00000800:00000200:12.0:1547251642.610197:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (16)++ 00000800:00000200:12.0:1547251642.610197:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (17)-- 00000800:00000200:12.0:1547251642.610198:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (16)-- 00000800:00000200:12.0:1547251642.610199:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (15)++ 00000800:00000200:12.0:1547251642.610200:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (16)-- 00000800:00000200:12.0:1547251642.610201:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (15)-- 00000800:00000200:15.0:1547251642.610202:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (14)++ 00000800:00000200:15.0:1547251642.610209:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (15)-- 00000800:00000200:15.0:1547251642.610210:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (14)-- 00000800:00000200:15.0:1547251642.610211:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (13)++ 00000800:00000200:15.0:1547251642.610214:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (14)-- 00000800:00000200:15.0:1547251642.610215:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (13)-- 00000800:00000200:15.0:1547251642.610216:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (12)++ 00000800:00000200:15.0:1547251642.610219:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (13)-- 00000800:00000200:15.0:1547251642.610220:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (12)-- 00000800:00000200:15.0:1547251642.610221:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (11)++ 00000800:00000200:15.0:1547251642.610222:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (12)-- 00000800:00000200:12.0:1547251642.610222:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (12)++ 00000800:00000200:15.0:1547251642.610223:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (12)-- 00000800:00000200:12.0:1547251642.610224:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (11)-- 00000800:00000200:15.0:1547251642.610225:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (11)++ 00000800:00000200:15.0:1547251642.610226:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (10)-- 00000800:00000200:12.0:1547251642.610226:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (11)-- 00000800:00000200:12.0:1547251642.610227:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (10)++ 00000800:00000200:15.0:1547251642.610228:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (10)-- 00000800:00000200:12.0:1547251642.610228:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (9)-- 00000800:00000200:15.0:1547251642.610229:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (9)++ 00000800:00000200:12.0:1547251642.610230:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (9)-- 00000800:00000200:15.0:1547251642.610231:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (8)-- 00000800:00000200:12.0:1547251642.610231:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (8)++ 00000800:00000200:15.0:1547251642.610232:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (8)-- 00000800:00000200:12.0:1547251642.610232:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (7)-- 00000800:00000200:15.0:1547251642.610233:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (7)++ 00000800:00000200:12.0:1547251642.610234:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (7)-- 00000800:00000200:15.0:1547251642.610235:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (6)-- 00000800:00000200:12.0:1547251642.610235:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (6)++ 00000800:00000200:15.0:1547251642.610236:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (6)-- 00000800:00000200:12.0:1547251642.610236:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (5)-- 00000800:00000200:15.0:1547251642.610237:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (5)++ 00000800:00000200:12.0:1547251642.610238:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (5)-- 00000800:00000200:15.0:1547251642.610239:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (4)-- 00000800:00000200:12.0:1547251642.610239:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (4)++ 00000800:00000200:15.0:1547251642.610241:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (4)-- 00000800:00000200:15.0:1547251642.610242:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (3)-- 00000800:00000200:12.0:1547251642.610242:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (3)-- 00000800:00000200:12.0:1547251642.610243:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (1)-- 00000800:00000200:9.0:1547251642.610255:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.610974:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251642.610979:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.610988:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251642.610989:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251642.610996:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251642.611004:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251642.611240:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251642.611247:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b028e4d000 0xbe8e4d000(0xbe8e4d000) 00000800:00000200:10.0:1547251642.611249:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97badbb75000 0x169bb75000(0x169bb75000) 00000800:00000200:10.0:1547251642.611250:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97a65d0ea000 0x21d0ea000(0x21d0ea000) 00000800:00000200:10.0:1547251642.611251:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ae3782d000 0x9f782d000(0x9f782d000) 00000800:00000200:10.0:1547251642.611251:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97a9952e3000 0x5552e3000(0x5552e3000) 00000800:00000200:10.0:1547251642.611252:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b81d517000 0x13dd517000(0x13dd517000) 00000800:00000200:10.0:1547251642.611253:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97baa1347000 0x1661347000(0x1661347000) 00000800:00000200:10.0:1547251642.611254:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b17b0b9000 0xd3b0b9000(0xd3b0b9000) 00000800:00000200:10.0:1547251642.611255:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b17b2d1000 0xd3b2d1000(0xd3b2d1000) 00000800:00000200:10.0:1547251642.611256:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97a65d3d9000 0x21d3d9000(0x21d3d9000) 00000800:00000200:10.0:1547251642.611256:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b824505000 0x13e4505000(0x13e4505000) 00000800:00000200:10.0:1547251642.611257:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97a99d3f9000 0x55d3f9000(0x55d3f9000) 00000800:00000200:10.0:1547251642.611258:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b824644000 0x13e4644000(0x13e4644000) 00000800:00000200:10.0:1547251642.611260:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ad4c32f000 0x90c32f000(0x90c32f000) 00000800:00000200:10.0:1547251642.611261:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b81d46c000 0x13dd46c000(0x13dd46c000) 00000800:00000200:10.0:1547251642.611262:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97acce225000 0x88e225000(0x88e225000) 00000800:00000200:10.0:1547251642.611263:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97a65d04d000 0x21d04d000(0x21d04d000) 00000800:00000200:10.0:1547251642.611264:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b81d79f000 0x13dd79f000(0x13dd79f000) 00000800:00000200:9.0:1547251642.611838:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611839:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611840:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611841:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611841:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611842:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611843:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611843:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611844:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611844:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611845:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611846:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611846:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611847:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611847:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611848:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611849:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611849:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611850:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611851:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611851:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611852:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611852:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611854:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611854:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611855:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611855:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611856:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611857:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611857:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611858:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611858:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611859:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611860:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.611860:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.611861:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000100:12.0:1547251642.612096:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251642.612098:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251642.612100:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251642.612141:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:12.0:1547251642.612154:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:15.0:1547251642.612193:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:15.0:1547251642.612203:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:15.0:1547251642.612204:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (19)-- 00000800:00000200:15.0:1547251642.612205:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (18)-- 00000800:00000200:8.2:1547251642.612290:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07fe00] (17)++ 00000800:00000200:15.0:1547251642.612303:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (18)++ 00000800:00000200:15.0:1547251642.612307:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (19)-- 00000800:00000200:15.0:1547251642.612308:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (18)-- 00000800:00000200:15.0:1547251642.612310:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (17)++ 00000800:00000200:15.0:1547251642.612311:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (18)-- 00000800:00000200:15.0:1547251642.612312:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (17)-- 00000800:00000200:15.0:1547251642.612313:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (16)++ 00000800:00000200:15.0:1547251642.612314:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (17)-- 00000800:00000200:15.0:1547251642.612315:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (16)-- 00000800:00000200:12.0:1547251642.612316:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (15)++ 00000800:00000200:12.0:1547251642.612320:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (16)-- 00000800:00000200:12.0:1547251642.612321:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (15)-- 00000800:00000200:12.0:1547251642.612321:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (14)++ 00000800:00000200:15.0:1547251642.612323:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (15)++ 00000800:00000200:12.0:1547251642.612323:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (15)-- 00000800:00000200:12.0:1547251642.612324:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (15)-- 00000800:00000200:15.0:1547251642.612325:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (14)-- 00000800:00000200:12.0:1547251642.612325:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (14)++ 00000800:00000200:15.0:1547251642.612326:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (14)-- 00000800:00000200:15.0:1547251642.612327:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (13)++ 00000800:00000200:12.0:1547251642.612327:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (13)-- 00000800:00000200:12.0:1547251642.612328:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (13)-- 00000800:00000200:15.0:1547251642.612329:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (12)-- 00000800:00000200:12.0:1547251642.612329:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (12)++ 00000800:00000200:15.0:1547251642.612330:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (12)-- 00000800:00000200:15.0:1547251642.612331:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (11)++ 00000800:00000200:12.0:1547251642.612331:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (11)-- 00000800:00000200:15.0:1547251642.612333:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (10)-- 00000800:00000200:12.0:1547251642.612333:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (11)-- 00000800:00000200:12.0:1547251642.612334:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (10)++ 00000800:00000200:15.0:1547251642.612335:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (10)-- 00000800:00000200:15.0:1547251642.612336:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (9)++ 00000800:00000200:12.0:1547251642.612336:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (9)-- 00000800:00000200:15.0:1547251642.612337:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (8)-- 00000800:00000200:12.0:1547251642.612337:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (9)-- 00000800:00000200:12.0:1547251642.612338:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (8)++ 00000800:00000200:15.0:1547251642.612339:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (8)-- 00000800:00000200:12.0:1547251642.612339:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (7)-- 00000800:00000200:15.0:1547251642.612340:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (7)++ 00000800:00000200:12.0:1547251642.612340:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (7)-- 00000800:00000200:15.0:1547251642.612341:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (6)-- 00000800:00000200:12.0:1547251642.612342:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (6)++ 00000800:00000200:15.0:1547251642.612343:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (6)-- 00000800:00000200:12.0:1547251642.612343:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (5)-- 00000800:00000200:15.0:1547251642.612344:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (5)++ 00000800:00000200:15.0:1547251642.612345:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (4)-- 00000800:00000200:12.0:1547251642.612345:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (5)-- 00000800:00000200:12.0:1547251642.612346:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (4)++ 00000800:00000200:15.0:1547251642.612347:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (4)-- 00000800:00000200:12.0:1547251642.612347:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (3)-- 00000800:00000200:15.0:1547251642.612348:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (3)-- 00000800:00000200:12.0:1547251642.612349:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (1)-- 00000800:00000200:9.0:1547251642.612359:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.612963:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251642.612967:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.612975:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251642.612976:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251642.612982:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251642.612987:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251642.613300:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251642.613309:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ae6a9fc000 0xa2a9fc000(0xa2a9fc000) 00000800:00000200:10.0:1547251642.613311:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba9ce49000 0x165ce49000(0x165ce49000) 00000800:00000200:10.0:1547251642.613312:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97bae1cfd000 0x16a1cfd000(0x16a1cfd000) 00000800:00000200:10.0:1547251642.613313:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ba90fac000 0x1650fac000(0x1650fac000) 00000800:00000200:10.0:1547251642.613313:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ac55dda000 0x815dda000(0x815dda000) 00000800:00000200:10.0:1547251642.613314:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ac55dd8000 0x815dd8000(0x815dd8000) 00000800:00000200:10.0:1547251642.613315:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b81dbcf000 0x13ddbcf000(0x13ddbcf000) 00000800:00000200:10.0:1547251642.613316:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b81d44f000 0x13dd44f000(0x13dd44f000) 00000800:00000200:10.0:1547251642.613317:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ba92af3000 0x1652af3000(0x1652af3000) 00000800:00000200:10.0:1547251642.613318:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97bae04f3000 0x16a04f3000(0x16a04f3000) 00000800:00000200:10.0:1547251642.613319:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ba9cf29000 0x165cf29000(0x165cf29000) 00000800:00000200:10.0:1547251642.613320:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97bae2635000 0x16a2635000(0x16a2635000) 00000800:00000200:10.0:1547251642.613321:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b82470c000 0x13e470c000(0x13e470c000) 00000800:00000200:10.0:1547251642.613321:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97babd0f1000 0x167d0f1000(0x167d0f1000) 00000800:00000200:10.0:1547251642.613322:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97bae01df000 0x16a01df000(0x16a01df000) 00000800:00000200:10.0:1547251642.613323:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba9f360000 0x165f360000(0x165f360000) 00000800:00000200:10.0:1547251642.613324:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b9073ab000 0x14c73ab000(0x14c73ab000) 00000800:00000200:10.0:1547251642.613325:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97bae4672000 0x16a4672000(0x16a4672000) 00000800:00000200:9.0:1547251642.613890:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613892:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613892:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613893:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613894:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613894:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613895:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613896:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613896:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613897:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613897:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613898:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613899:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613899:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613900:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613900:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613901:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613902:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613902:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613903:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613904:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613905:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613905:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613906:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613906:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613907:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613908:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613908:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613909:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613909:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613910:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613911:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613911:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613912:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.613912:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.613913:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000100:12.0:1547251642.614144:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251642.614146:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251642.614147:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251642.614185:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:12.0:1547251642.614197:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:15.0:1547251642.614236:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:15.0:1547251642.614244:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:15.0:1547251642.614244:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (19)-- 00000800:00000200:15.0:1547251642.614246:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (18)-- 00000800:00000200:8.2:1547251642.614326:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07dc00] (17)++ 00000800:00000200:12.0:1547251642.614339:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (18)++ 00000800:00000200:12.0:1547251642.614344:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (19)-- 00000800:00000200:12.0:1547251642.614345:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (18)-- 00000800:00000200:12.0:1547251642.614346:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (17)++ 00000800:00000200:12.0:1547251642.614347:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (18)-- 00000800:00000200:12.0:1547251642.614348:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (17)-- 00000800:00000200:12.0:1547251642.614348:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (16)++ 00000800:00000200:12.0:1547251642.614349:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (17)-- 00000800:00000200:12.0:1547251642.614350:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (16)-- 00000800:00000200:12.0:1547251642.614350:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (15)++ 00000800:00000200:15.0:1547251642.614352:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (16)++ 00000800:00000200:12.0:1547251642.614352:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (16)-- 00000800:00000200:12.0:1547251642.614353:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (16)-- 00000800:00000200:15.0:1547251642.614354:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (15)-- 00000800:00000200:12.0:1547251642.614354:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (15)++ 00000800:00000200:15.0:1547251642.614355:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (15)-- 00000800:00000200:15.0:1547251642.614356:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (14)++ 00000800:00000200:12.0:1547251642.614356:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (14)-- 00000800:00000200:12.0:1547251642.614357:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (14)-- 00000800:00000200:15.0:1547251642.614358:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (13)-- 00000800:00000200:12.0:1547251642.614358:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (13)++ 00000800:00000200:15.0:1547251642.614359:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (13)-- 00000800:00000200:15.0:1547251642.614360:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (12)++ 00000800:00000200:12.0:1547251642.614360:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (12)-- 00000800:00000200:12.0:1547251642.614361:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (12)-- 00000800:00000200:15.0:1547251642.614362:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (11)-- 00000800:00000200:12.0:1547251642.614362:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (11)++ 00000800:00000200:12.0:1547251642.614363:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (12)-- 00000800:00000200:12.0:1547251642.614364:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (11)-- 00000800:00000200:15.0:1547251642.614365:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (9)-- 00000800:00000200:12.0:1547251642.614366:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (8)++ 00000800:00000200:12.0:1547251642.614369:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (9)-- 00000800:00000200:12.0:1547251642.614370:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (8)-- 00000800:00000200:12.0:1547251642.614371:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (7)++ 00000800:00000200:15.0:1547251642.614372:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (8)++ 00000800:00000200:12.0:1547251642.614372:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (8)-- 00000800:00000200:12.0:1547251642.614373:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (8)-- 00000800:00000200:15.0:1547251642.614374:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (7)-- 00000800:00000200:12.0:1547251642.614374:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (7)++ 00000800:00000200:15.0:1547251642.614375:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (7)-- 00000800:00000200:15.0:1547251642.614376:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (6)++ 00000800:00000200:12.0:1547251642.614376:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (6)-- 00000800:00000200:12.0:1547251642.614377:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (6)-- 00000800:00000200:15.0:1547251642.614378:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (5)-- 00000800:00000200:12.0:1547251642.614378:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (5)++ 00000800:00000200:15.0:1547251642.614379:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (5)-- 00000800:00000200:15.0:1547251642.614380:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (4)++ 00000800:00000200:12.0:1547251642.614380:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (4)-- 00000800:00000200:15.0:1547251642.614382:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (3)-- 00000800:00000200:12.0:1547251642.614382:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (4)-- 00000800:00000200:12.0:1547251642.614383:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (3)-- 00000800:00000200:15.0:1547251642.614384:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (1)-- 00000800:00000200:9.0:1547251642.614396:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.615139:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251642.615142:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.615151:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251642.615152:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251642.615157:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251642.615163:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251642.615401:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251642.615409:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba9bae9000 0x165bae9000(0x165bae9000) 00000800:00000200:10.0:1547251642.615410:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b6268c2000 0x11e68c2000(0x11e68c2000) 00000800:00000200:10.0:1547251642.615412:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97badf0ab000 0x169f0ab000(0x169f0ab000) 00000800:00000200:10.0:1547251642.615413:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97badc84d000 0x169c84d000(0x169c84d000) 00000800:00000200:10.0:1547251642.615414:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97aeead1f000 0xaaad1f000(0xaaad1f000) 00000800:00000200:10.0:1547251642.615415:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba9bf33000 0x165bf33000(0x165bf33000) 00000800:00000200:10.0:1547251642.615416:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97baa0e2c000 0x1660e2c000(0x1660e2c000) 00000800:00000200:10.0:1547251642.615417:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97babd0e4000 0x167d0e4000(0x167d0e4000) 00000800:00000200:10.0:1547251642.615417:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97a9951ba000 0x5551ba000(0x5551ba000) 00000800:00000200:10.0:1547251642.615418:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97bae1342000 0x16a1342000(0x16a1342000) 00000800:00000200:10.0:1547251642.615419:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97a8c76a9000 0x4876a9000(0x4876a9000) 00000800:00000200:10.0:1547251642.615420:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ae37822000 0x9f7822000(0x9f7822000) 00000800:00000200:10.0:1547251642.615421:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba9da6f000 0x165da6f000(0x165da6f000) 00000800:00000200:10.0:1547251642.615422:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97baddebc000 0x169debc000(0x169debc000) 00000800:00000200:10.0:1547251642.615423:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba9ba4b000 0x165ba4b000(0x165ba4b000) 00000800:00000200:10.0:1547251642.615423:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba9bf9c000 0x165bf9c000(0x165bf9c000) 00000800:00000200:10.0:1547251642.615424:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97badd522000 0x169d522000(0x169d522000) 00000800:00000200:10.0:1547251642.615425:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97bad6a8c000 0x1696a8c000(0x1696a8c000) 00000800:00000200:9.0:1547251642.615995:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.615996:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.615997:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.615997:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.615998:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.615999:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.615999:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616000:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616000:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616001:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616002:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616002:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616003:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616004:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616004:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616005:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616006:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616006:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616007:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616008:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616008:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616009:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616009:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616010:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616011:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616011:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616012:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616012:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616013:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616014:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616014:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616015:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616015:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616016:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.616017:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.616017:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000100:12.0:1547251642.616248:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251642.616251:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251642.616252:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251642.616302:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:12.0:1547251642.616312:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:12.0:1547251642.616316:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:12.0:1547251642.616321:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:12.0:1547251642.616321:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (19)-- 00000800:00000200:12.0:1547251642.616322:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (18)-- 00000800:00000200:8.2:1547251642.616438:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f600] (17)++ 00000800:00000200:12.0:1547251642.616449:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (18)++ 00000800:00000200:12.0:1547251642.616453:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (19)-- 00000800:00000200:12.0:1547251642.616454:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (18)-- 00000800:00000200:12.0:1547251642.616455:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (17)++ 00000800:00000200:12.0:1547251642.616455:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (18)-- 00000800:00000200:12.0:1547251642.616456:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (17)-- 00000800:00000200:12.0:1547251642.616457:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (16)++ 00000800:00000200:12.0:1547251642.616458:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (17)-- 00000800:00000200:12.0:1547251642.616458:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (16)-- 00000800:00000200:12.0:1547251642.616459:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (15)++ 00000800:00000200:12.0:1547251642.616460:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (16)-- 00000800:00000200:12.0:1547251642.616461:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (15)-- 00000800:00000200:15.0:1547251642.616462:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (14)++ 00000800:00000200:15.0:1547251642.616469:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (15)-- 00000800:00000200:15.0:1547251642.616470:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (14)-- 00000800:00000200:15.0:1547251642.616471:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (13)++ 00000800:00000200:15.0:1547251642.616474:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (14)-- 00000800:00000200:15.0:1547251642.616475:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (13)-- 00000800:00000200:15.0:1547251642.616476:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (12)++ 00000800:00000200:15.0:1547251642.616477:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (13)-- 00000800:00000200:12.0:1547251642.616477:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (13)++ 00000800:00000200:15.0:1547251642.616478:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (13)-- 00000800:00000200:15.0:1547251642.616479:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (12)++ 00000800:00000200:12.0:1547251642.616479:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (12)-- 00000800:00000200:12.0:1547251642.616480:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (12)-- 00000800:00000200:15.0:1547251642.616481:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (11)-- 00000800:00000200:12.0:1547251642.616481:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (11)++ 00000800:00000200:15.0:1547251642.616482:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (11)-- 00000800:00000200:15.0:1547251642.616483:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (10)++ 00000800:00000200:12.0:1547251642.616483:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (10)-- 00000800:00000200:12.0:1547251642.616484:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (10)-- 00000800:00000200:15.0:1547251642.616485:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (9)-- 00000800:00000200:12.0:1547251642.616485:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (9)++ 00000800:00000200:15.0:1547251642.616486:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (9)-- 00000800:00000200:15.0:1547251642.616487:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (8)++ 00000800:00000200:12.0:1547251642.616487:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (8)-- 00000800:00000200:12.0:1547251642.616488:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (8)-- 00000800:00000200:15.0:1547251642.616489:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (7)-- 00000800:00000200:12.0:1547251642.616489:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (7)++ 00000800:00000200:15.0:1547251642.616490:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (7)-- 00000800:00000200:15.0:1547251642.616491:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (6)++ 00000800:00000200:12.0:1547251642.616491:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (6)-- 00000800:00000200:12.0:1547251642.616492:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (6)-- 00000800:00000200:15.0:1547251642.616493:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (5)-- 00000800:00000200:12.0:1547251642.616493:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (5)++ 00000800:00000200:15.0:1547251642.616494:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (5)-- 00000800:00000200:15.0:1547251642.616495:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (4)++ 00000800:00000200:12.0:1547251642.616495:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (4)-- 00000800:00000200:12.0:1547251642.616496:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (4)-- 00000800:00000200:15.0:1547251642.616497:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (3)-- 00000800:00000200:12.0:1547251642.616497:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (3)-- 00000800:00000200:15.0:1547251642.616498:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (1)-- 00000800:00000200:9.0:1547251642.616510:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.617116:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251642.617120:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.617128:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251642.617129:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251642.617135:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251642.617140:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251642.617371:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251642.617377:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba967dc000 0x16567dc000(0x16567dc000) 00000800:00000200:10.0:1547251642.617378:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97baa0e16000 0x1660e16000(0x1660e16000) 00000800:00000200:10.0:1547251642.617379:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba970b3000 0x16570b3000(0x16570b3000) 00000800:00000200:10.0:1547251642.617379:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97bae1d46000 0x16a1d46000(0x16a1d46000) 00000800:00000200:10.0:1547251642.617380:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b80cd6d000 0x13ccd6d000(0x13ccd6d000) 00000800:00000200:10.0:1547251642.617381:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba9a7f7000 0x165a7f7000(0x165a7f7000) 00000800:00000200:10.0:1547251642.617382:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ba8fc21000 0x164fc21000(0x164fc21000) 00000800:00000200:10.0:1547251642.617383:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97babd302000 0x167d302000(0x167d302000) 00000800:00000200:10.0:1547251642.617384:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ba8de94000 0x164de94000(0x164de94000) 00000800:00000200:10.0:1547251642.617384:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97baa13a5000 0x16613a5000(0x16613a5000) 00000800:00000200:10.0:1547251642.617385:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ae5c43e000 0xa1c43e000(0xa1c43e000) 00000800:00000200:10.0:1547251642.617386:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97babd5d3000 0x167d5d3000(0x167d5d3000) 00000800:00000200:10.0:1547251642.617388:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b139d01000 0xcf9d01000(0xcf9d01000) 00000800:00000200:10.0:1547251642.617389:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97a765da6000 0x325da6000(0x325da6000) 00000800:00000200:10.0:1547251642.617389:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97babd481000 0x167d481000(0x167d481000) 00000800:00000200:10.0:1547251642.617391:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97bade7f9000 0x169e7f9000(0x169e7f9000) 00000800:00000200:10.0:1547251642.617391:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b6268ce000 0x11e68ce000(0x11e68ce000) 00000800:00000200:10.0:1547251642.617392:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97badd67c000 0x169d67c000(0x169d67c000) 00000800:00000200:9.0:1547251642.617951:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617953:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617953:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617954:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617955:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617955:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617956:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617957:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617957:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617958:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617958:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617959:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617960:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617961:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617961:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617962:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617963:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617963:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617964:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617964:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617965:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617966:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617966:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617967:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617967:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617968:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617969:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617969:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617970:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617970:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617971:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617972:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617972:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617973:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251642.617973:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251642.617974:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000100:12.0:1547251642.618257:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251642.618260:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251642.618261:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251642.618308:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:12.0:1547251642.618314:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:12.0:1547251642.618318:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:12.0:1547251642.618322:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:12.0:1547251642.618323:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (19)-- 00000800:00000200:12.0:1547251642.618324:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (18)-- 00000800:00000200:8.2:1547251642.618445:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f600] (17)++ 00000800:00000200:12.0:1547251642.618456:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (18)++ 00000800:00000200:12.0:1547251642.618461:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (19)-- 00000800:00000200:12.0:1547251642.618461:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (18)-- 00000800:00000200:12.0:1547251642.618462:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (17)++ 00000800:00000200:12.0:1547251642.618463:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (18)-- 00000800:00000200:12.0:1547251642.618464:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (17)-- 00000800:00000200:12.0:1547251642.618465:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (16)++ 00000800:00000200:12.0:1547251642.618465:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (17)-- 00000800:00000200:12.0:1547251642.618466:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (16)-- 00000800:00000200:12.0:1547251642.618467:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (15)++ 00000800:00000200:12.0:1547251642.618468:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (16)-- 00000800:00000200:12.0:1547251642.618469:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (15)-- 00000800:00000200:15.0:1547251642.618470:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (14)++ 00000800:00000200:15.0:1547251642.618477:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (15)-- 00000800:00000200:15.0:1547251642.618478:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (14)-- 00000800:00000200:12.0:1547251642.618481:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (13)++ 00000800:00000200:12.0:1547251642.618484:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (14)-- 00000800:00000200:12.0:1547251642.618484:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (13)-- 00000800:00000200:12.0:1547251642.618485:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (12)++ 00000800:00000200:12.0:1547251642.618486:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (13)-- 00000800:00000200:15.0:1547251642.618487:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (13)++ 00000800:00000200:12.0:1547251642.618488:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (13)-- 00000800:00000200:15.0:1547251642.618489:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (12)-- 00000800:00000200:12.0:1547251642.618489:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (12)++ 00000800:00000200:15.0:1547251642.618490:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (12)-- 00000800:00000200:15.0:1547251642.618491:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (11)++ 00000800:00000200:12.0:1547251642.618491:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (11)-- 00000800:00000200:12.0:1547251642.618492:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (11)-- 00000800:00000200:15.0:1547251642.618493:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (10)-- 00000800:00000200:12.0:1547251642.618493:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (10)++ 00000800:00000200:15.0:1547251642.618494:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (10)-- 00000800:00000200:15.0:1547251642.618495:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (9)++ 00000800:00000200:12.0:1547251642.618495:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (9)-- 00000800:00000200:12.0:1547251642.618496:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (9)-- 00000800:00000200:15.0:1547251642.618497:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (8)-- 00000800:00000200:12.0:1547251642.618497:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (8)++ 00000800:00000200:15.0:1547251642.618498:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (8)-- 00000800:00000200:12.0:1547251642.618499:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (7)-- 00000800:00000200:15.0:1547251642.618500:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (7)++ 00000800:00000200:12.0:1547251642.618500:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (7)-- 00000800:00000200:15.0:1547251642.618501:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (6)-- 00000800:00000200:12.0:1547251642.618502:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (6)++ 00000800:00000200:15.0:1547251642.618503:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (6)-- 00000800:00000200:12.0:1547251642.618503:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (5)-- 00000800:00000200:15.0:1547251642.618504:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (5)++ 00000800:00000200:15.0:1547251642.618505:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (4)-- 00000800:00000200:12.0:1547251642.618505:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (5)-- 00000800:00000200:15.0:1547251642.618506:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (4)-- 00000800:00000200:12.0:1547251642.618506:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (4)++ 00000800:00000200:12.0:1547251642.618507:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (3)-- 00000800:00000200:15.0:1547251642.618508:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (3)-- 00000800:00000200:12.0:1547251642.618509:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (1)-- 00000800:00000200:9.0:1547251642.618519:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.619441:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251642.619445:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.619455:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251642.619456:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251642.619462:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251642.619471:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251642.619723:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251642.619732:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b46b417000 0x102b417000(0x102b417000) 00000800:00000200:10.0:1547251642.619734:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ae5d2c5000 0xa1d2c5000(0xa1d2c5000) 00000800:00000200:10.0:1547251642.619735:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b17b1c6000 0xd3b1c6000(0xd3b1c6000) 00000800:00000200:10.0:1547251642.619736:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97a84d7c5000 0x40d7c5000(0x40d7c5000) 00000800:00000200:10.0:1547251642.619737:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b81d79f000 0x13dd79f000(0x13dd79f000) 00000800:00000200:10.0:1547251642.619738:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97bae17eb000 0x16a17eb000(0x16a17eb000) 00000800:00000200:10.0:1547251642.619739:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97a954a67000 0x514a67000(0x514a67000) 00000800:00000200:10.0:1547251642.619740:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba8f26a000 0x164f26a000(0x164f26a000) 00000800:00000200:10.0:1547251642.619740:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ba9e2da000 0x165e2da000(0x165e2da000) 00000800:00000200:10.0:1547251642.619741:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97a5c2ccd000 0x182ccd000(0x182ccd000) 00000800:00000200:10.0:1547251642.619742:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ae6a85e000 0xa2a85e000(0xa2a85e000) 00000800:00000200:10.0:1547251642.619743:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97bae2735000 0x16a2735000(0x16a2735000) 00000800:00000200:10.0:1547251642.619744:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba974d1000 0x16574d1000(0x16574d1000) 00000800:00000200:10.0:1547251642.619745:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba941ef000 0x16541ef000(0x16541ef000) 00000800:00000200:10.0:1547251642.619746:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b8244d9000 0x13e44d9000(0x13e44d9000) 00000800:00000200:10.0:1547251642.619746:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97badc8ee000 0x169c8ee000(0x169c8ee000) 00000800:00000200:10.0:1547251642.619747:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b8245cf000 0x13e45cf000(0x13e45cf000) 00000800:00000200:10.0:1547251642.619748:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba9c23e000 0x165c23e000(0x165c23e000) 00000800:00000200:9.0:1547251642.620320:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620321:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620322:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620323:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620323:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620324:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620325:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620326:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620326:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620327:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620328:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620328:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620329:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620330:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620330:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620331:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620331:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620332:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620333:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620333:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620334:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620335:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620335:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620336:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620336:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620337:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620338:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620338:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620339:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620339:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620340:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620341:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620341:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620342:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:9.0:1547251642.620342:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:9.0:1547251642.620343:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000100:12.0:1547251642.620613:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251642.620616:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251642.620617:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251642.620669:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:12.0:1547251642.620678:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:15.0:1547251642.620721:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (19)++ 00000800:00000200:15.0:1547251642.620729:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (20)-- 00000800:00000200:15.0:1547251642.620730:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (19)-- 00000800:00000200:15.0:1547251642.620732:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (18)-- 00000800:00000200:8.2:1547251642.620809:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07dc00] (17)++ 00000800:00000200:15.0:1547251642.620821:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (18)++ 00000800:00000200:15.0:1547251642.620826:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (19)-- 00000800:00000200:15.0:1547251642.620826:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (18)-- 00000800:00000200:15.0:1547251642.620827:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (17)++ 00000800:00000200:15.0:1547251642.620828:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (18)-- 00000800:00000200:15.0:1547251642.620828:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (17)-- 00000800:00000200:15.0:1547251642.620829:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (16)++ 00000800:00000200:15.0:1547251642.620830:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (17)-- 00000800:00000200:15.0:1547251642.620831:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (16)-- 00000800:00000200:15.0:1547251642.620831:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (15)++ 00000800:00000200:15.0:1547251642.620835:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (16)-- 00000800:00000200:15.0:1547251642.620835:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (15)-- 00000800:00000200:15.0:1547251642.620836:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (14)++ 00000800:00000200:15.0:1547251642.620839:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (15)-- 00000800:00000200:15.0:1547251642.620840:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (14)-- 00000800:00000200:15.0:1547251642.620841:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (13)++ 00000800:00000200:15.0:1547251642.620843:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (14)-- 00000800:00000200:15.0:1547251642.620845:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (13)-- 00000800:00000200:12.0:1547251642.620846:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (12)++ 00000800:00000200:12.0:1547251642.620851:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (13)-- 00000800:00000200:12.0:1547251642.620852:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (12)-- 00000800:00000200:12.0:1547251642.620853:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (11)++ 00000800:00000200:12.0:1547251642.620856:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (12)-- 00000800:00000200:12.0:1547251642.620856:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (11)-- 00000800:00000200:12.0:1547251642.620857:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (10)++ 00000800:00000200:15.0:1547251642.620859:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (11)++ 00000800:00000200:12.0:1547251642.620859:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (11)-- 00000800:00000200:12.0:1547251642.620860:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (11)-- 00000800:00000200:15.0:1547251642.620861:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (10)-- 00000800:00000200:12.0:1547251642.620861:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (10)++ 00000800:00000200:15.0:1547251642.620862:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (10)-- 00000800:00000200:15.0:1547251642.620863:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (9)++ 00000800:00000200:12.0:1547251642.620863:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (9)-- 00000800:00000200:12.0:1547251642.620864:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (9)-- 00000800:00000200:15.0:1547251642.620865:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (8)-- 00000800:00000200:12.0:1547251642.620865:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (8)++ 00000800:00000200:15.0:1547251642.620866:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (8)-- 00000800:00000200:15.0:1547251642.620867:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (7)++ 00000800:00000200:12.0:1547251642.620867:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (7)-- 00000800:00000200:12.0:1547251642.620868:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (7)-- 00000800:00000200:15.0:1547251642.620869:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (6)-- 00000800:00000200:12.0:1547251642.620869:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (6)++ 00000800:00000200:15.0:1547251642.620870:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (6)-- 00000800:00000200:15.0:1547251642.620871:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (5)++ 00000800:00000200:12.0:1547251642.620871:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (5)-- 00000800:00000200:12.0:1547251642.620872:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (5)-- 00000800:00000200:15.0:1547251642.620873:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (4)-- 00000800:00000200:12.0:1547251642.620873:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07dc00] (4)++ 00000800:00000200:15.0:1547251642.620874:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (4)-- 00000800:00000200:15.0:1547251642.620875:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (3)-- 00000800:00000200:12.0:1547251642.620875:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07dc00] (3)-- 00000800:00000200:12.0:1547251642.620876:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07dc00] (1)-- 00000800:00000200:9.0:1547251642.620887:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.621613:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251642.621617:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.621627:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251642.621628:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251642.621636:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251642.621644:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251642.621876:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251642.621885:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97babd50e000 0x167d50e000(0x167d50e000) 00000800:00000200:10.0:1547251642.621886:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba9ffcc000 0x165ffcc000(0x165ffcc000) 00000800:00000200:10.0:1547251642.621887:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba9d3ea000 0x165d3ea000(0x165d3ea000) 00000800:00000200:10.0:1547251642.621888:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97baa0e56000 0x1660e56000(0x1660e56000) 00000800:00000200:10.0:1547251642.621889:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97baa0e9e000 0x1660e9e000(0x1660e9e000) 00000800:00000200:10.0:1547251642.621889:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97bae273e000 0x16a273e000(0x16a273e000) 00000800:00000200:10.0:1547251642.621890:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97bae1bf4000 0x16a1bf4000(0x16a1bf4000) 00000800:00000200:10.0:1547251642.621891:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97bade64f000 0x169e64f000(0x169e64f000) 00000800:00000200:10.0:1547251642.621892:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ba927fa000 0x16527fa000(0x16527fa000) 00000800:00000200:10.0:1547251642.621893:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba9a6df000 0x165a6df000(0x165a6df000) 00000800:00000200:10.0:1547251642.621894:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b17b056000 0xd3b056000(0xd3b056000) 00000800:00000200:10.0:1547251642.621895:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b5bdd35000 0x117dd35000(0x117dd35000) 00000800:00000200:10.0:1547251642.621895:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ae6ab19000 0xa2ab19000(0xa2ab19000) 00000800:00000200:10.0:1547251642.621896:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b81d76f000 0x13dd76f000(0x13dd76f000) 00000800:00000200:10.0:1547251642.621897:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b139e8e000 0xcf9e8e000(0xcf9e8e000) 00000800:00000200:10.0:1547251642.621898:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b67c47a000 0x123c47a000(0x123c47a000) 00000800:00000200:10.0:1547251642.621899:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ac55dda000 0x815dda000(0x815dda000) 00000800:00000200:10.0:1547251642.621900:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ac55dd8000 0x815dd8000(0x815dd8000) 00000800:00000200:9.0:1547251642.622520:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622522:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622522:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622524:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622524:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622525:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622525:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622526:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622527:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622527:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622528:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622529:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622529:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622530:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622530:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622531:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622532:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622532:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622533:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622534:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622534:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622535:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622535:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622536:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622537:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622537:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622538:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622538:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622539:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622540:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622540:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622541:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622541:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622542:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251642.622543:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251642.622543:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000100:12.0:1547251642.622812:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251642.622815:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251642.622816:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251642.622865:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:12.0:1547251642.622867:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:15.0:1547251642.622916:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:15.0:1547251642.622925:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:15.0:1547251642.622925:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (19)-- 00000800:00000200:15.0:1547251642.622927:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (18)-- 00000800:00000200:8.2:1547251642.623000:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07fe00] (17)++ 00000800:00000200:15.0:1547251642.623012:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (18)++ 00000800:00000200:15.0:1547251642.623016:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (19)-- 00000800:00000200:15.0:1547251642.623017:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (18)-- 00000800:00000200:15.0:1547251642.623018:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (17)++ 00000800:00000200:15.0:1547251642.623018:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (18)-- 00000800:00000200:15.0:1547251642.623019:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (17)-- 00000800:00000200:15.0:1547251642.623020:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (16)++ 00000800:00000200:15.0:1547251642.623021:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (17)-- 00000800:00000200:15.0:1547251642.623022:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (16)-- 00000800:00000200:12.0:1547251642.623023:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (15)++ 00000800:00000200:12.0:1547251642.623027:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (16)-- 00000800:00000200:12.0:1547251642.623028:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (15)-- 00000800:00000200:12.0:1547251642.623029:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (14)++ 00000800:00000200:15.0:1547251642.623030:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (15)++ 00000800:00000200:12.0:1547251642.623030:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (15)-- 00000800:00000200:15.0:1547251642.623031:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (16)-- 00000800:00000200:15.0:1547251642.623032:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (14)-- 00000800:00000200:15.0:1547251642.623033:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (12)++ 00000800:00000200:12.0:1547251642.623033:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (13)-- 00000800:00000200:15.0:1547251642.623037:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (13)-- 00000800:00000200:15.0:1547251642.623038:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (12)-- 00000800:00000200:15.0:1547251642.623039:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (11)++ 00000800:00000200:15.0:1547251642.623040:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (12)-- 00000800:00000200:12.0:1547251642.623040:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (12)++ 00000800:00000200:15.0:1547251642.623041:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (12)-- 00000800:00000200:12.0:1547251642.623042:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (11)-- 00000800:00000200:15.0:1547251642.623043:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (11)++ 00000800:00000200:12.0:1547251642.623045:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (11)-- 00000800:00000200:15.0:1547251642.623046:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (10)-- 00000800:00000200:12.0:1547251642.623046:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (10)++ 00000800:00000200:15.0:1547251642.623047:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (10)-- 00000800:00000200:15.0:1547251642.623048:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (9)++ 00000800:00000200:12.0:1547251642.623048:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (9)-- 00000800:00000200:12.0:1547251642.623049:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (9)-- 00000800:00000200:15.0:1547251642.623050:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (8)-- 00000800:00000200:12.0:1547251642.623050:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (8)++ 00000800:00000200:15.0:1547251642.623051:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (8)-- 00000800:00000200:15.0:1547251642.623052:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (7)++ 00000800:00000200:12.0:1547251642.623052:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (7)-- 00000800:00000200:12.0:1547251642.623054:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (7)-- 00000800:00000200:15.0:1547251642.623055:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (6)-- 00000800:00000200:12.0:1547251642.623055:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (6)++ 00000800:00000200:15.0:1547251642.623056:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (6)-- 00000800:00000200:12.0:1547251642.623056:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (5)-- 00000800:00000200:15.0:1547251642.623057:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (5)++ 00000800:00000200:15.0:1547251642.623058:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (4)-- 00000800:00000200:12.0:1547251642.623058:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (5)-- 00000800:00000200:15.0:1547251642.623059:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (4)-- 00000800:00000200:12.0:1547251642.623059:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (4)++ 00000800:00000200:12.0:1547251642.623060:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (3)-- 00000800:00000200:15.0:1547251642.623061:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (3)-- 00000800:00000200:12.0:1547251642.623062:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (1)-- 00000800:00000200:9.0:1547251642.623073:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.623800:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251642.623805:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.623814:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251642.623816:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251642.623821:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251642.623828:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251642.624057:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251642.624062:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba92af3000 0x1652af3000(0x1652af3000) 00000800:00000200:10.0:1547251642.624063:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97bae04f3000 0x16a04f3000(0x16a04f3000) 00000800:00000200:10.0:1547251642.624064:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba9cf29000 0x165cf29000(0x165cf29000) 00000800:00000200:10.0:1547251642.624065:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97bae2635000 0x16a2635000(0x16a2635000) 00000800:00000200:10.0:1547251642.624066:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b82470c000 0x13e470c000(0x13e470c000) 00000800:00000200:10.0:1547251642.624067:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97babd0f1000 0x167d0f1000(0x167d0f1000) 00000800:00000200:10.0:1547251642.624068:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97bae01df000 0x16a01df000(0x16a01df000) 00000800:00000200:10.0:1547251642.624069:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba9f360000 0x165f360000(0x165f360000) 00000800:00000200:10.0:1547251642.624069:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b9073ab000 0x14c73ab000(0x14c73ab000) 00000800:00000200:10.0:1547251642.624070:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97bae4672000 0x16a4672000(0x16a4672000) 00000800:00000200:10.0:1547251642.624071:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b028e4d000 0xbe8e4d000(0xbe8e4d000) 00000800:00000200:10.0:1547251642.624073:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97badbb75000 0x169bb75000(0x169bb75000) 00000800:00000200:10.0:1547251642.624074:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97baa1347000 0x1661347000(0x1661347000) 00000800:00000200:10.0:1547251642.624075:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97a65d3d9000 0x21d3d9000(0x21d3d9000) 00000800:00000200:10.0:1547251642.624076:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97a99d3f9000 0x55d3f9000(0x55d3f9000) 00000800:00000200:10.0:1547251642.624077:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b824644000 0x13e4644000(0x13e4644000) 00000800:00000200:10.0:1547251642.624078:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97bae1a95000 0x16a1a95000(0x16a1a95000) 00000800:00000200:10.0:1547251642.624078:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba912e1000 0x16512e1000(0x16512e1000) 00000800:00000200:9.0:1547251642.624654:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624655:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624656:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624656:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624657:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624657:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624658:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624659:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624659:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624660:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624660:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624661:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624662:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624662:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624663:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624664:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624664:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624665:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624665:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624666:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624667:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624667:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624668:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624668:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624669:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624670:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624670:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624671:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624671:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624672:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624673:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624673:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624674:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624675:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251642.624675:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251642.624676:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000100:12.0:1547251642.624907:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251642.624909:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251642.624910:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:12.0:1547251642.624954:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c600] (19)-- 00000800:00000200:8.2:1547251642.624964:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c600] (18)++ 00000800:00000200:15.0:1547251642.625016:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:15.0:1547251642.625025:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:15.0:1547251642.625026:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (19)-- 00000800:00000200:15.0:1547251642.625027:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:8.2:1547251642.625101:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c600] (17)++ 00000800:00000200:15.0:1547251642.625114:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)++ 00000800:00000200:15.0:1547251642.625118:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (19)-- 00000800:00000200:15.0:1547251642.625119:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:15.0:1547251642.625120:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (17)++ 00000800:00000200:15.0:1547251642.625121:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:15.0:1547251642.625121:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (17)-- 00000800:00000200:15.0:1547251642.625122:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (16)++ 00000800:00000200:15.0:1547251642.625123:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (17)-- 00000800:00000200:15.0:1547251642.625124:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (16)-- 00000800:00000200:12.0:1547251642.625125:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)++ 00000800:00000200:12.0:1547251642.625129:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (16)-- 00000800:00000200:12.0:1547251642.625130:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)-- 00000800:00000200:12.0:1547251642.625131:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (14)++ 00000800:00000200:15.0:1547251642.625132:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)++ 00000800:00000200:12.0:1547251642.625132:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (15)-- 00000800:00000200:12.0:1547251642.625133:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)-- 00000800:00000200:15.0:1547251642.625134:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (14)-- 00000800:00000200:12.0:1547251642.625134:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (14)++ 00000800:00000200:15.0:1547251642.625135:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (14)-- 00000800:00000200:12.0:1547251642.625136:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:15.0:1547251642.625137:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)++ 00000800:00000200:12.0:1547251642.625138:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:15.0:1547251642.625139:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (12)-- 00000800:00000200:12.0:1547251642.625139:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)++ 00000800:00000200:15.0:1547251642.625140:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)-- 00000800:00000200:15.0:1547251642.625141:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)++ 00000800:00000200:12.0:1547251642.625141:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (11)-- 00000800:00000200:12.0:1547251642.625142:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)-- 00000800:00000200:15.0:1547251642.625143:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (10)-- 00000800:00000200:12.0:1547251642.625143:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (10)++ 00000800:00000200:15.0:1547251642.625144:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (10)-- 00000800:00000200:15.0:1547251642.625145:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (9)++ 00000800:00000200:12.0:1547251642.625145:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (9)-- 00000800:00000200:12.0:1547251642.625146:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (9)-- 00000800:00000200:15.0:1547251642.625147:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:12.0:1547251642.625147:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)++ 00000800:00000200:15.0:1547251642.625149:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:15.0:1547251642.625152:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (7)++ 00000800:00000200:12.0:1547251642.625152:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (7)-- 00000800:00000200:12.0:1547251642.625153:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (7)-- 00000800:00000200:15.0:1547251642.625154:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (6)-- 00000800:00000200:12.0:1547251642.625154:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (6)++ 00000800:00000200:15.0:1547251642.625155:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (6)-- 00000800:00000200:15.0:1547251642.625156:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)++ 00000800:00000200:12.0:1547251642.625156:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (5)-- 00000800:00000200:12.0:1547251642.625157:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)-- 00000800:00000200:15.0:1547251642.625158:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (4)-- 00000800:00000200:12.0:1547251642.625158:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (4)++ 00000800:00000200:15.0:1547251642.625159:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (4)-- 00000800:00000200:12.0:1547251642.625160:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (3)-- 00000800:00000200:15.0:1547251642.625161:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (3)-- 00000800:00000200:12.0:1547251642.625161:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (1)-- 00000800:00000200:9.0:1547251642.625173:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.625813:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251642.625816:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.625824:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251642.625826:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251642.625832:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251642.625839:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251642.626018:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251642.626024:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba95c64000 0x1655c64000(0x1655c64000) 00000800:00000200:10.0:1547251642.626025:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97bade77b000 0x169e77b000(0x169e77b000) 00000800:00000200:10.0:1547251642.626026:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba957c3000 0x16557c3000(0x16557c3000) 00000800:00000200:10.0:1547251642.626027:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b1231bf000 0xce31bf000(0xce31bf000) 00000800:00000200:10.0:1547251642.626028:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ba95ba2000 0x1655ba2000(0x1655ba2000) 00000800:00000200:10.0:1547251642.626029:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba99fe5000 0x1659fe5000(0x1659fe5000) 00000800:00000200:10.0:1547251642.626030:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ba94159000 0x1654159000(0x1654159000) 00000800:00000200:10.0:1547251642.626031:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97baa13a5000 0x16613a5000(0x16613a5000) 00000800:00000200:10.0:1547251642.626032:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ae5c43e000 0xa1c43e000(0xa1c43e000) 00000800:00000200:10.0:1547251642.626033:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97babd5d3000 0x167d5d3000(0x167d5d3000) 00000800:00000200:10.0:1547251642.626033:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b139d01000 0xcf9d01000(0xcf9d01000) 00000800:00000200:10.0:1547251642.626034:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97a765da6000 0x325da6000(0x325da6000) 00000800:00000200:10.0:1547251642.626035:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97babd481000 0x167d481000(0x167d481000) 00000800:00000200:10.0:1547251642.626036:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97bade7f9000 0x169e7f9000(0x169e7f9000) 00000800:00000200:10.0:1547251642.626037:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b6268ce000 0x11e68ce000(0x11e68ce000) 00000800:00000200:10.0:1547251642.626038:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97badd67c000 0x169d67c000(0x169d67c000) 00000800:00000200:10.0:1547251642.626039:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ba9bae9000 0x165bae9000(0x165bae9000) 00000800:00000200:10.0:1547251642.626039:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b6268c2000 0x11e68c2000(0x11e68c2000) 00000800:00000200:9.0:1547251642.626582:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626583:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626591:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626591:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626592:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626593:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626593:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626594:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626595:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626595:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626596:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626596:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626597:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626598:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626598:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626599:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626600:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626600:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626601:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626601:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626602:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626603:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626603:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626604:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626604:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626605:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626606:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626606:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626607:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626608:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626608:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626609:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626610:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626610:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251642.626611:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251642.626612:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000100:12.0:1547251642.626862:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251642.626865:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251642.626866:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251642.626928:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:12.0:1547251642.626938:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:15.0:1547251642.626979:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:15.0:1547251642.626986:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:15.0:1547251642.626987:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (19)-- 00000800:00000200:15.0:1547251642.626989:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (18)-- 00000800:00000200:8.2:1547251642.627012:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c800] (17)++ 00000800:00000200:15.0:1547251642.627025:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (18)++ 00000800:00000200:15.0:1547251642.627028:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (19)-- 00000800:00000200:15.0:1547251642.627029:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (18)-- 00000800:00000200:15.0:1547251642.627030:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (17)++ 00000800:00000200:15.0:1547251642.627031:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (18)-- 00000800:00000200:15.0:1547251642.627032:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (17)-- 00000800:00000200:12.0:1547251642.627033:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (16)++ 00000800:00000200:12.0:1547251642.627037:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (17)-- 00000800:00000200:12.0:1547251642.627038:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (16)-- 00000800:00000200:12.0:1547251642.627039:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (15)++ 00000800:00000200:15.0:1547251642.627040:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (16)++ 00000800:00000200:12.0:1547251642.627040:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (16)-- 00000800:00000200:12.0:1547251642.627041:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (16)-- 00000800:00000200:15.0:1547251642.627042:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (15)-- 00000800:00000200:12.0:1547251642.627042:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (15)++ 00000800:00000200:15.0:1547251642.627043:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (15)-- 00000800:00000200:15.0:1547251642.627044:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (14)++ 00000800:00000200:12.0:1547251642.627044:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (14)-- 00000800:00000200:12.0:1547251642.627045:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (14)-- 00000800:00000200:15.0:1547251642.627046:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (13)-- 00000800:00000200:15.0:1547251642.627047:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (13)-- 00000800:00000200:12.0:1547251642.627047:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (13)++ 00000800:00000200:12.0:1547251642.627048:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (12)-- 00000800:00000200:15.0:1547251642.627049:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (12)++ 00000800:00000200:12.0:1547251642.627050:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (12)-- 00000800:00000200:15.0:1547251642.627051:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (11)-- 00000800:00000200:12.0:1547251642.627051:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (11)++ 00000800:00000200:15.0:1547251642.627052:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (11)-- 00000800:00000200:12.0:1547251642.627052:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (10)-- 00000800:00000200:15.0:1547251642.627053:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (10)++ 00000800:00000200:12.0:1547251642.627054:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (10)-- 00000800:00000200:15.0:1547251642.627055:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (9)-- 00000800:00000200:12.0:1547251642.627055:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (9)++ 00000800:00000200:15.0:1547251642.627056:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (9)-- 00000800:00000200:15.0:1547251642.627057:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (8)++ 00000800:00000200:12.0:1547251642.627057:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (8)-- 00000800:00000200:15.0:1547251642.627058:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (7)-- 00000800:00000200:12.0:1547251642.627058:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (8)-- 00000800:00000200:12.0:1547251642.627059:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (7)++ 00000800:00000200:15.0:1547251642.627060:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (7)-- 00000800:00000200:15.0:1547251642.627061:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (6)++ 00000800:00000200:12.0:1547251642.627061:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (6)-- 00000800:00000200:12.0:1547251642.627062:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (6)-- 00000800:00000200:15.0:1547251642.627063:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (5)-- 00000800:00000200:12.0:1547251642.627063:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (5)++ 00000800:00000200:15.0:1547251642.627064:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (5)-- 00000800:00000200:12.0:1547251642.627064:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (4)-- 00000800:00000200:15.0:1547251642.627065:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (4)++ 00000800:00000200:12.0:1547251642.627066:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (4)-- 00000800:00000200:15.0:1547251642.627068:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (3)-- 00000800:00000200:12.0:1547251642.627068:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (3)-- 00000800:00000200:15.0:1547251642.627070:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (1)-- 00000800:00000200:9.0:1547251642.627081:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251642.627863:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000100:00000200:8.0:1547251642.698680:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446320, portal 4 00000100:00000200:8.0:1547251642.698682:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446320, offset 0 00000400:00000200:8.0:1547251642.698685:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251642.698688:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251642.698690:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23063] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:2.0:1547251642.698692:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446336, portal 4 00000400:00000200:8.0:1547251642.698694:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23061] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23063] 00000100:00000200:2.0:1547251642.698695:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446336, offset 0 00000400:00000200:8.0:1547251642.698696:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251642.698697:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251642.698698:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251642.698699:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251642.698700:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9754, 9754] 00000400:00000200:8.0:1547251642.698702:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:2.0:1547251642.698702:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:2.0:1547251642.698705:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23064] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251642.698707:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:2.0:1547251642.698708:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23061] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23064] 00000800:00000200:8.0:1547251642.698710:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:2.0:1547251642.698710:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251642.698712:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251642.698713:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:2.0:1547251642.698713:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251642.698714:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:8.0:1547251642.698715:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000400:00000200:2.0:1547251642.698715:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9754, 9755] 00000400:00000200:2.0:1547251642.698716:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:8.0:1547251642.698718:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000400:00000200:2.0:1547251642.698721:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251642.698724:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:2.0:1547251642.698727:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251642.698728:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:2.0:1547251642.698729:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:2.0:1547251642.698731:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:12.2:1547251642.698741:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000100:00000200:13.0:1547251642.698752:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446352, portal 4 00000800:00000200:0.2:1547251642.698753:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000100:00000200:13.0:1547251642.698755:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446352, offset 0 00000400:00000200:13.0:1547251642.698759:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:13.0:1547251642.698763:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:13.0:1547251642.698766:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23064] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:13.0:1547251642.698775:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:254, d:10, s:23062] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23064] 00000400:00000200:13.0:1547251642.698777:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:13.0:1547251642.698778:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251642.698780:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251642.698781:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9755, 9755] 00000400:00000200:13.0:1547251642.698783:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:13.0:1547251642.698788:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251642.698790:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:13.0:1547251642.698793:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251642.698795:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (22)++ 00000800:00000200:13.0:1547251642.698796:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00000800:00000200:13.0:1547251642.698799:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (24)-- 00000800:00000200:5.0:1547251642.698803:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:5.0:1547251642.698807:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000800:00000200:4.0:1547251642.698807:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251642.698809:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:5.0:1547251642.698811:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824751500 x1622384012446320/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251642.698813:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f5100 x1622384012446336/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251642.698819:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688aa0 00000400:00000200:4.0:1547251642.698819:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a5d8 00000400:00000200:5.0:1547251642.698820:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251642.698820:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251642.698824:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251642.698824:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251642.698827:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251642.698828:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251642.698829:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251642.698830:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:5.0:1547251642.698831:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251642.698832:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:5.0:1547251642.698833:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a0000 x1622384012446352/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251642.698839:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b990 00000400:00000200:5.0:1547251642.698840:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251642.698843:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251642.698845:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:0.2:1547251642.698891:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:12.2:1547251642.698904:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:4.0:1547251642.698910:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251642.698915:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000800:00000200:5.0:1547251642.698918:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000400:00000200:4.0:1547251642.698919:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:5.0:1547251642.698921:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251642.698924:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04a70 00000400:00000200:5.0:1547251642.698925:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251642.698929:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04a80 00000400:00000200:4.0:1547251642.698929:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4015 [1] + 192 00000400:00000200:5.0:1547251642.698933:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4025 [1] + 192 00000100:00000200:4.0:1547251642.698937:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824751500 x1622384012446320/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251642.698958:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251642.698961:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000100:00000200:5.0:1547251642.698962:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f5100 x1622384012446336/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251642.698963:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:4.0:1547251642.698965:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251642.698967:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251642.698970:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251642.698974:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251642.698975:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251642.698977:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824751500 x1622384012446320/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251642.698977:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251642.698978:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000400:00000200:4.0:1547251642.698978:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04a90 00000800:00000200:5.0:1547251642.698980:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251642.698981:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000100:00000200:8.0:1547251642.698982:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824751500 x1622384012446320/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251642.698983:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:8.0:1547251642.698986:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00010000:00080000:8.0:1547251642.698990:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824751500 x1622384012446320/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:4.0:1547251642.698990:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4035 [1] + 192 00000100:00000200:2.0:1547251642.698995:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f5100 x1622384012446336/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251642.699001:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f5100 x1622384012446336/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251642.699005:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a6e8 00000100:00000200:4.0:1547251642.699008:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a0000 x1622384012446352/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:2.0:1547251642.699008:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f5100 x1622384012446336/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:4.0:1547251642.699017:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251642.699018:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251642.699020:0:12661:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251642.699022:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:4.0:1547251642.699023:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000100:00000200:13.0:1547251642.699027:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a0000 x1622384012446352/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:13.0:1547251642.699033:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a0000 x1622384012446352/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:0.2:1547251642.699035:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000400:00000200:13.0:1547251642.699037:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b2a8 00010000:00080000:13.0:1547251642.699040:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a0000 x1622384012446352/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251649 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:5.0:1547251642.699048:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:5.0:1547251642.699053:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:5.0:1547251642.699055:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251642.699056:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000100:00000200:8.0:1547251643.098681:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446368, portal 4 00000100:00000200:8.0:1547251643.098684:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446368, offset 0 00000400:00000200:8.0:1547251643.098686:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251643.098689:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251643.098692:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04aa0 00000400:00000200:8.0:1547251643.098694:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4999 [1] + 0 00000400:00000200:8.0:1547251643.098696:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ba9d367c38 00000100:00000200:8.0:1547251643.098698:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251643.098706:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba9d367c38 00000400:00000200:8.0:1547251643.098707:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251643.098711:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824751500 x1622384012446368/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251650 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251643.098717:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446384, portal 4 00000400:00000200:8.0:1547251643.098718:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688dd0 00000400:00000200:8.0:1547251643.098719:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251643.098720:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251643.098720:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446384, offset 0 00000400:00000200:4.0:1547251643.098723:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251643.098726:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:9.0:1547251643.098728:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446368 00000400:00000200:4.0:1547251643.098729:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04ab0 00000400:00000200:4.0:1547251643.098731:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe28d5 [1] + 0 00000400:00000200:4.0:1547251643.098733:0:12676:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040b4c8 00000100:00000200:4.0:1547251643.098735:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00010000:00000200:9.0:1547251643.098738:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3d9c50 x1622384012446368/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:704/0 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:4.0:1547251643.098743:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b4c8 00000400:00000200:4.0:1547251643.098744:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251643.098745:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446368, offset 192 00000100:00000200:4.0:1547251643.098746:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c430000 x1622384012446384/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251650 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251643.098747:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251643.098750:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251643.098751:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bee0 00000400:00000200:9.0:1547251643.098752:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04aa0 00000400:00000200:4.0:1547251643.098752:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251643.098753:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:9.0:1547251643.098755:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf1e1 [1] + 192 00000100:00000200:9.0:1547251643.098757:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824751500 x1622384012446368/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251650 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251643.098766:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251643.098767:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446384 00000400:00000200:9.0:1547251643.098768:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea220 00000400:00000200:9.0:1547251643.098769:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251643.098771:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251643.098775:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824751500 x1622384012446368/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251650 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251643.098777:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00010000:00000200:6.0:1547251643.098780:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d399450 x1622384012446384/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:704/0 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:8.0:1547251643.098781:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824751500 x1622384012446368/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251650 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251643.098787:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000100:00000200:6.0:1547251643.098789:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446384, offset 192 00010000:00080000:8.0:1547251643.098791:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824751500 x1622384012446368/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251650 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:6.0:1547251643.098793:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251643.098796:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251643.098799:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04ab0 00000400:00000200:6.0:1547251643.098802:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf1f1 [1] + 192 00000100:00000200:6.0:1547251643.098805:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c430000 x1622384012446384/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251650 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251643.098815:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251643.098817:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990330 00000400:00000200:6.0:1547251643.098818:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251643.098819:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251643.098824:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c430000 x1622384012446384/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251650 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251643.098825:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:4.0:1547251643.098828:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c430000 x1622384012446384/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251650 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251643.098832:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bc38 00010000:00080000:4.0:1547251643.098835:0:12676:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae5c430000 x1622384012446384/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251650 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251643.114685:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446400, portal 4 00000100:00000200:4.0:1547251643.114688:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446400, offset 0 00000400:00000200:4.0:1547251643.114690:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251643.114693:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251643.114696:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04ac0 00000400:00000200:4.0:1547251643.114699:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2905 [1] + 0 00000400:00000200:4.0:1547251643.114701:0:12675:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040bcc0 00000100:00000200:4.0:1547251643.114703:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:4.0:1547251643.114711:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bcc0 00000400:00000200:4.0:1547251643.114712:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251643.114714:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c434500 x1622384012446400/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251650 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251643.114719:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bee0 00000400:00000200:4.0:1547251643.114720:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251643.114721:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251643.114733:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446400 00010000:00000200:6.0:1547251643.114745:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d399050 x1622384012446400/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:704/0 lens 224/368 e 0 to 0 dl 1547251649 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251643.114755:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446400, offset 192 00000400:00000200:6.0:1547251643.114758:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251643.114762:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251643.114765:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04ac0 00000400:00000200:6.0:1547251643.114768:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf201 [1] + 192 00000100:00000200:6.0:1547251643.114771:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c434500 x1622384012446400/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251650 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251643.114781:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251643.114783:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990f68 00000400:00000200:6.0:1547251643.114784:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251643.114785:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251643.114789:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c434500 x1622384012446400/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251650 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251643.114790:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:4.0:1547251643.114794:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c434500 x1622384012446400/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251650 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251643.114798:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bc38 00010000:00080000:4.0:1547251643.114801:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae5c434500 x1622384012446400/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251650 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:9.0:1547251643.307666:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251643.307680:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251643.307682:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251643.307689:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251643.307698:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251643.307922:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251643.307928:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97aeead1f000 0xaaad1f000(0xaaad1f000) 00000800:00000200:10.0:1547251643.307930:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba9bf33000 0x165bf33000(0x165bf33000) 00000800:00000200:10.0:1547251643.307931:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97baa0e2c000 0x1660e2c000(0x1660e2c000) 00000800:00000200:10.0:1547251643.307932:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97babd0e4000 0x167d0e4000(0x167d0e4000) 00000800:00000200:10.0:1547251643.307933:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97a9951ba000 0x5551ba000(0x5551ba000) 00000800:00000200:10.0:1547251643.307934:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97bae1342000 0x16a1342000(0x16a1342000) 00000800:00000200:10.0:1547251643.307935:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97a8c76a9000 0x4876a9000(0x4876a9000) 00000800:00000200:10.0:1547251643.307935:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ae37822000 0x9f7822000(0x9f7822000) 00000800:00000200:10.0:1547251643.307936:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ba9da6f000 0x165da6f000(0x165da6f000) 00000800:00000200:10.0:1547251643.307937:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97baddebc000 0x169debc000(0x169debc000) 00000800:00000200:10.0:1547251643.307938:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ba9ba4b000 0x165ba4b000(0x165ba4b000) 00000800:00000200:10.0:1547251643.307939:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba9bf9c000 0x165bf9c000(0x165bf9c000) 00000800:00000200:10.0:1547251643.307940:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97badd522000 0x169d522000(0x169d522000) 00000800:00000200:10.0:1547251643.307940:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97bad6a8c000 0x1696a8c000(0x1696a8c000) 00000800:00000200:10.0:1547251643.307948:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba9ce49000 0x165ce49000(0x165ce49000) 00000800:00000200:10.0:1547251643.307949:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97bae1cfd000 0x16a1cfd000(0x16a1cfd000) 00000800:00000200:10.0:1547251643.307950:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ba90fac000 0x1650fac000(0x1650fac000) 00000800:00000200:10.0:1547251643.307951:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97a601aa3000 0x1c1aa3000(0x1c1aa3000) 00000800:00000200:9.0:1547251643.308639:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308641:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308642:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308643:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308643:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308644:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308645:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308645:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308646:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308647:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308647:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308648:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308649:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308650:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308650:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308651:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308651:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308652:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308653:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308653:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308654:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308654:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308655:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308656:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308656:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308657:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308658:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308658:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308659:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308659:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308660:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308661:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308661:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308662:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251643.308662:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251643.308663:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000100:12.0:1547251643.308952:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251643.308955:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251643.308956:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251643.308993:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:12.0:1547251643.309004:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (20)++ 00000800:00000200:12.0:1547251643.309009:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (21)-- 00000800:00000200:12.0:1547251643.309010:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:12.0:1547251643.309011:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (19)-- 00000800:00000200:12.0:1547251643.309021:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c400] (18)-- 00000800:00000200:8.2:1547251643.309128:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c400] (17)++ 00000800:00000200:12.0:1547251643.309139:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (18)++ 00000800:00000200:12.0:1547251643.309144:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (19)-- 00000800:00000200:12.0:1547251643.309144:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (18)-- 00000800:00000200:12.0:1547251643.309145:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (17)++ 00000800:00000200:12.0:1547251643.309146:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (18)-- 00000800:00000200:12.0:1547251643.309147:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (17)-- 00000800:00000200:12.0:1547251643.309148:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (16)++ 00000800:00000200:12.0:1547251643.309149:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (17)-- 00000800:00000200:12.0:1547251643.309149:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (16)-- 00000800:00000200:12.0:1547251643.309150:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (15)++ 00000800:00000200:12.0:1547251643.309153:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (16)-- 00000800:00000200:12.0:1547251643.309154:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (15)-- 00000800:00000200:12.0:1547251643.309155:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (14)++ 00000800:00000200:12.0:1547251643.309156:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (15)-- 00000800:00000200:12.0:1547251643.309157:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (14)-- 00000800:00000200:15.0:1547251643.309158:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (13)++ 00000800:00000200:15.0:1547251643.309165:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (14)-- 00000800:00000200:15.0:1547251643.309166:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (13)-- 00000800:00000200:15.0:1547251643.309167:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (12)++ 00000800:00000200:15.0:1547251643.309170:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (13)-- 00000800:00000200:15.0:1547251643.309170:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (12)-- 00000800:00000200:15.0:1547251643.309171:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (11)++ 00000800:00000200:15.0:1547251643.309173:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (12)-- 00000800:00000200:12.0:1547251643.309173:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (12)++ 00000800:00000200:15.0:1547251643.309174:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (12)-- 00000800:00000200:15.0:1547251643.309175:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (11)++ 00000800:00000200:12.0:1547251643.309175:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (11)-- 00000800:00000200:12.0:1547251643.309176:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (11)-- 00000800:00000200:15.0:1547251643.309177:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (10)-- 00000800:00000200:12.0:1547251643.309177:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (10)++ 00000800:00000200:15.0:1547251643.309178:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (10)-- 00000800:00000200:15.0:1547251643.309179:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (9)++ 00000800:00000200:12.0:1547251643.309179:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (9)-- 00000800:00000200:12.0:1547251643.309180:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (9)-- 00000800:00000200:15.0:1547251643.309181:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (8)-- 00000800:00000200:12.0:1547251643.309181:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (8)++ 00000800:00000200:15.0:1547251643.309182:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (8)-- 00000800:00000200:15.0:1547251643.309183:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (7)++ 00000800:00000200:12.0:1547251643.309183:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (7)-- 00000800:00000200:12.0:1547251643.309184:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (7)-- 00000800:00000200:15.0:1547251643.309185:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (6)-- 00000800:00000200:12.0:1547251643.309185:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (6)++ 00000800:00000200:15.0:1547251643.309186:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (6)-- 00000800:00000200:15.0:1547251643.309187:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (5)++ 00000800:00000200:12.0:1547251643.309187:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (5)-- 00000800:00000200:12.0:1547251643.309188:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (5)-- 00000800:00000200:15.0:1547251643.309190:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (4)-- 00000800:00000200:12.0:1547251643.309190:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (4)++ 00000800:00000200:15.0:1547251643.309191:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (4)-- 00000800:00000200:12.0:1547251643.309192:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (3)-- 00000800:00000200:15.0:1547251643.309193:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (3)-- 00000800:00000200:12.0:1547251643.309194:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (1)-- 00000800:00000200:9.0:1547251643.309206:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251643.309856:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251644.307665:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251644.307678:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251644.307680:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251644.307688:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251644.307697:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251644.308270:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251644.308277:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba92b9a000 0x1652b9a000(0x1652b9a000) 00000800:00000200:10.0:1547251644.308278:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b81d7dc000 0x13dd7dc000(0x13dd7dc000) 00000800:00000200:10.0:1547251644.308279:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b1233d3000 0xce33d3000(0xce33d3000) 00000800:00000200:10.0:1547251644.308280:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ae6a988000 0xa2a988000(0xa2a988000) 00000800:00000200:10.0:1547251644.308281:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ae6a915000 0xa2a915000(0xa2a915000) 00000800:00000200:10.0:1547251644.308282:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97baa285e000 0x166285e000(0x166285e000) 00000800:00000200:10.0:1547251644.308283:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b028e16000 0xbe8e16000(0xbe8e16000) 00000800:00000200:10.0:1547251644.308284:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97a601bab000 0x1c1bab000(0x1c1bab000) 00000800:00000200:10.0:1547251644.308285:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97a995224000 0x555224000(0x555224000) 00000800:00000200:10.0:1547251644.308285:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ae3787e000 0x9f787e000(0x9f787e000) 00000800:00000200:10.0:1547251644.308287:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97a601b82000 0x1c1b82000(0x1c1b82000) 00000800:00000200:10.0:1547251644.308288:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97baa0f30000 0x1660f30000(0x1660f30000) 00000800:00000200:10.0:1547251644.308289:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba91a4b000 0x1651a4b000(0x1651a4b000) 00000800:00000200:10.0:1547251644.308290:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba961ae000 0x16561ae000(0x16561ae000) 00000800:00000200:10.0:1547251644.308291:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b80cd1b000 0x13ccd1b000(0x13ccd1b000) 00000800:00000200:10.0:1547251644.308292:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97baa1dbf000 0x1661dbf000(0x1661dbf000) 00000800:00000200:10.0:1547251644.308293:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97badc8bf000 0x169c8bf000(0x169c8bf000) 00000800:00000200:10.0:1547251644.308294:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba92fc0000 0x1652fc0000(0x1652fc0000) 00000800:00000200:9.0:1547251644.308966:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308968:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308969:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308970:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308970:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308971:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308971:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308973:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308973:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308974:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308975:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308975:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308976:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308976:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308977:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308978:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308978:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308979:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308980:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308980:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308981:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308981:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308982:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308983:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308983:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308984:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308984:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308985:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308986:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308986:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308987:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308987:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308988:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308989:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251644.308989:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251644.308990:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000100:12.0:1547251644.309532:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251644.309536:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251644.309537:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251644.309574:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:12.0:1547251644.309599:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:15.0:1547251644.309630:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:15.0:1547251644.309639:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:15.0:1547251644.309640:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (19)-- 00000800:00000200:15.0:1547251644.309641:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (18)-- 00000800:00000200:8.2:1547251644.309712:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f200] (17)++ 00000800:00000200:15.0:1547251644.309724:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (18)++ 00000800:00000200:15.0:1547251644.309728:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (19)-- 00000800:00000200:15.0:1547251644.309729:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (18)-- 00000800:00000200:15.0:1547251644.309730:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (17)++ 00000800:00000200:15.0:1547251644.309731:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (18)-- 00000800:00000200:15.0:1547251644.309731:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (17)-- 00000800:00000200:15.0:1547251644.309732:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (16)++ 00000800:00000200:15.0:1547251644.309733:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (17)-- 00000800:00000200:15.0:1547251644.309733:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (16)-- 00000800:00000200:15.0:1547251644.309734:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (15)++ 00000800:00000200:15.0:1547251644.309739:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (16)-- 00000800:00000200:15.0:1547251644.309739:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (15)-- 00000800:00000200:15.0:1547251644.309740:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (14)++ 00000800:00000200:15.0:1547251644.309743:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (15)-- 00000800:00000200:15.0:1547251644.309744:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (14)-- 00000800:00000200:15.0:1547251644.309745:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (13)++ 00000800:00000200:15.0:1547251644.309748:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (14)-- 00000800:00000200:15.0:1547251644.309748:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (13)-- 00000800:00000200:15.0:1547251644.309749:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (12)++ 00000800:00000200:15.0:1547251644.309752:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (13)-- 00000800:00000200:15.0:1547251644.309753:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (12)-- 00000800:00000200:15.0:1547251644.309754:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (11)++ 00000800:00000200:15.0:1547251644.309756:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (12)-- 00000800:00000200:15.0:1547251644.309757:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (11)-- 00000800:00000200:15.0:1547251644.309758:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (10)++ 00000800:00000200:15.0:1547251644.309761:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (11)-- 00000800:00000200:15.0:1547251644.309761:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (10)-- 00000800:00000200:15.0:1547251644.309762:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (9)++ 00000800:00000200:15.0:1547251644.309765:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (10)-- 00000800:00000200:15.0:1547251644.309766:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (9)-- 00000800:00000200:15.0:1547251644.309767:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (8)++ 00000800:00000200:15.0:1547251644.309769:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (9)-- 00000800:00000200:15.0:1547251644.309770:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (8)-- 00000800:00000200:15.0:1547251644.309771:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (7)++ 00000800:00000200:15.0:1547251644.309774:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (8)-- 00000800:00000200:15.0:1547251644.309775:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (7)-- 00000800:00000200:15.0:1547251644.309775:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (6)++ 00000800:00000200:15.0:1547251644.309777:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (7)-- 00000800:00000200:15.0:1547251644.309777:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (6)-- 00000800:00000200:12.0:1547251644.309778:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (5)++ 00000800:00000200:12.0:1547251644.309783:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (6)-- 00000800:00000200:12.0:1547251644.309784:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (5)-- 00000800:00000200:12.0:1547251644.309785:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (4)++ 00000800:00000200:12.0:1547251644.309788:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (5)-- 00000800:00000200:12.0:1547251644.309788:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (4)-- 00000800:00000200:12.0:1547251644.309789:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (3)++ 00000800:00000200:12.0:1547251644.309790:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (4)-- 00000800:00000200:15.0:1547251644.309791:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (4)++ 00000800:00000200:12.0:1547251644.309792:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (4)-- 00000800:00000200:15.0:1547251644.309793:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (3)-- 00000800:00000200:12.0:1547251644.309793:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (3)-- 00000800:00000200:15.0:1547251644.309794:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (1)-- 00000800:00000200:9.0:1547251644.309809:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251644.310724:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000100:00000200:11.0:1547251645.130680:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446416, portal 4 00000100:00000200:11.0:1547251645.130684:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446416, offset 0 00000400:00000200:11.0:1547251645.130688:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:11.0:1547251645.130691:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:11.0:1547251645.130695:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23065] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:11.0:1547251645.130699:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23062] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23065] 00000400:00000200:11.0:1547251645.130703:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:11.0:1547251645.130705:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251645.130706:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251645.130707:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9755, 9757] 00000400:00000200:11.0:1547251645.130709:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:11.0:1547251645.130714:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251645.130716:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:11.0:1547251645.130719:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251645.130721:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:11.0:1547251645.130722:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:11.0:1547251645.130725:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:4.0:1547251645.130744:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446448, portal 4 00000100:00000200:4.0:1547251645.130747:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446448, offset 0 00000100:00000200:0.0:1547251645.130748:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446432, portal 4 00000800:00000200:7.2:1547251645.130749:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:4.0:1547251645.130750:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000100:00000200:0.0:1547251645.130751:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446432, offset 0 00000400:00000200:4.0:1547251645.130753:0:12675:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:0.0:1547251645.130754:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:4.0:1547251645.130756:0:12675:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23066] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251645.130757:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:4.0:1547251645.130759:0:12675:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23062] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23066] 00000400:00000200:0.0:1547251645.130760:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23066] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:4.0:1547251645.130761:0:12675:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:4.0:1547251645.130763:0:12675:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251645.130763:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23063] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23066] 00000400:00000200:0.0:1547251645.130765:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:12.0:1547251645.130766:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000400:00000200:4.0:1547251645.130766:0:12675:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251645.130767:0:12675:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9756, 9757] 00000400:00000200:0.0:1547251645.130767:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251645.130768:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251645.130769:0:12675:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:0.0:1547251645.130769:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9756, 9757] 00000400:00000200:0.0:1547251645.130771:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:4.0:1547251645.130774:0:12675:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:4.0:1547251645.130777:0:12675:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:12.0:1547251645.130778:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:4.0:1547251645.130780:0:12675:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000400:00000200:0.0:1547251645.130780:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:4.0:1547251645.130781:0:12675:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:4.0:1547251645.130782:0:12675:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000100:00000200:12.0:1547251645.130783:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07cb00 x1622384012446416/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251652 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:0.0:1547251645.130783:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:15.0:1547251645.130785:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:4.0:1547251645.130785:0:12675:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:0.0:1547251645.130786:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251645.130789:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:12.0:1547251645.130790:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bf68 00000800:00000200:0.0:1547251645.130790:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (22)++ 00000400:00000200:12.0:1547251645.130792:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:0.0:1547251645.130793:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (23)-- 00000400:00000200:12.0:1547251645.130795:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251645.130797:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:9.2:1547251645.130801:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:15.0:1547251645.130815:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:15.0:1547251645.130823:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (24)-- 00000100:00000200:15.0:1547251645.130827:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df4800 x1622384012446448/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251652 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251645.130830:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:12.0:1547251645.130831:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (24)-- 00000100:00000200:12.0:1547251645.130834:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f4e00 x1622384012446432/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251652 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251645.130837:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bee0 00000400:00000200:12.0:1547251645.130838:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665000 00000400:00000200:15.0:1547251645.130839:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251645.130839:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251645.130841:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:15.0:1547251645.130842:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251645.130843:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:15.0:1547251645.130844:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251645.130845:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:9.2:1547251645.130914:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.2:1547251645.130919:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:7.2:1547251645.130925:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:15.0:1547251645.130926:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251645.130929:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.93@o2ib10 00000400:00000200:15.0:1547251645.130932:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251645.130932:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251645.130934:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251645.130936:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04af0 00000400:00000200:12.0:1547251645.130937:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251645.130939:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fddd [1] + 192 00000400:00000200:12.0:1547251645.130948:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04ad0 00000100:00000200:15.0:1547251645.130951:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df4800 x1622384012446448/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251652 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251645.130960:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251645.130962:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000400:00000200:12.0:1547251645.130962:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fdcd [1] + 192 00000800:00000200:15.0:1547251645.130964:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:15.0:1547251645.130964:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:12.0:1547251645.130965:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07cb00 x1622384012446416/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251652 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251645.130966:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:15.0:1547251645.130968:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:15.0:1547251645.130971:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251645.130974:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04ae0 00000400:00000200:12.0:1547251645.130974:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251645.130976:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000100:00000200:4.0:1547251645.130976:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df4800 x1622384012446448/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251652 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251645.130978:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251645.130979:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251645.130980:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:12.0:1547251645.130981:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000100:00000200:4.0:1547251645.130982:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df4800 x1622384012446448/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251652 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251645.130983:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000400:00000200:4.0:1547251645.130986:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bc38 00000400:00000200:15.0:1547251645.130988:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fded [1] + 192 00010000:00080000:4.0:1547251645.130990:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a765df4800 x1622384012446448/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251652 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251645.130992:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07cb00 x1622384012446416/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251652 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251645.130998:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07cb00 x1622384012446416/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251652 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251645.131002:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b908 00000100:00000200:15.0:1547251645.131005:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f4e00 x1622384012446432/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251652 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:11.0:1547251645.131007:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b17b07cb00 x1622384012446416/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251652 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:15.0:1547251645.131013:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251645.131014:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (21)++ 00000800:00000200:15.0:1547251645.131016:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (22)-- 00000800:00000200:15.0:1547251645.131016:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:0.0:1547251645.131022:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f4e00 x1622384012446432/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251652 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251645.131028:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f4e00 x1622384012446432/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251652 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251645.131032:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00010000:00080000:0.0:1547251645.131036:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f4e00 x1622384012446432/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251652 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:9.0:1547251645.307661:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251645.307674:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251645.307676:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000400:9.0:1547251645.307681:0:12658:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.2.10.184@o2ib10: 0 seconds 00000800:00000200:10.0:1547251645.307683:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000400:00000200:9.0:1547251645.307686:0:12658:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = -110, hstatus = 6 00000400:00000200:9.0:1547251645.307689:0:12658:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.2.10.184@o2ib10: PUT: LOCAL_TIMEOUT 00000800:00000200:10.0:1547251645.307693:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000400:00000100:9.0:1547251645.307693:0:12658:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.2.10.184@o2ib10 exceeded retry count 0 00000400:00000200:9.0:1547251645.307695:0:12658:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06ab28 00000800:00000200:10.0:1547251645.308202:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251645.308211:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba95590000 0x1655590000(0x1655590000) 00000800:00000200:10.0:1547251645.308213:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba92952000 0x1652952000(0x1652952000) 00000800:00000200:10.0:1547251645.308214:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba927e7000 0x16527e7000(0x16527e7000) 00000800:00000200:10.0:1547251645.308214:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ba8f6ca000 0x164f6ca000(0x164f6ca000) 00000800:00000200:10.0:1547251645.308215:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97badcaf8000 0x169caf8000(0x169caf8000) 00000800:00000200:10.0:1547251645.308216:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba9b621000 0x165b621000(0x165b621000) 00000800:00000200:10.0:1547251645.308217:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ba9c7ad000 0x165c7ad000(0x165c7ad000) 00000800:00000200:10.0:1547251645.308218:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97badc8d2000 0x169c8d2000(0x169c8d2000) 00000800:00000200:10.0:1547251645.308219:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ba92997000 0x1652997000(0x1652997000) 00000800:00000200:10.0:1547251645.308220:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97bae13b0000 0x16a13b0000(0x16a13b0000) 00000800:00000200:10.0:1547251645.308221:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ba92af3000 0x1652af3000(0x1652af3000) 00000800:00000200:10.0:1547251645.308221:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97bae04f3000 0x16a04f3000(0x16a04f3000) 00000800:00000200:10.0:1547251645.308222:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba9cf29000 0x165cf29000(0x165cf29000) 00000800:00000200:10.0:1547251645.308223:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97bae2635000 0x16a2635000(0x16a2635000) 00000800:00000200:10.0:1547251645.308224:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b82470c000 0x13e470c000(0x13e470c000) 00000800:00000200:10.0:1547251645.308225:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97babd0f1000 0x167d0f1000(0x167d0f1000) 00000800:00000200:10.0:1547251645.308226:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97bae01df000 0x16a01df000(0x16a01df000) 00000800:00000200:10.0:1547251645.308227:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba9f360000 0x165f360000(0x165f360000) 00000800:00000200:9.0:1547251645.308910:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308912:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308913:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308914:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308915:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308915:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308916:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308917:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308917:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308918:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308918:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308919:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308920:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308920:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308921:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308922:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308922:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308923:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308924:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308924:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308925:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308925:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308926:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308927:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308927:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308928:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308929:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308929:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308930:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308930:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308931:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308932:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308932:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308933:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251645.308933:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251645.308935:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000100:12.0:1547251645.309223:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: don't reconnect (no need), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251645.309226:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251645.309228:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:12.0:1547251645.309230:0:70:0:(o2iblnd.c:417:kiblnd_unlink_peer_locked()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)-- 00000400:00000200:12.0:1547251645.309234:0:70:0:(router.c:1756:lnet_notify()) 10.1.11.90@o2ib10 notifying 10.2.10.184@o2ib10: down 00000400:00000200:12.0:1547251645.309235:0:70:0:(router.c:120:lnet_notify_locked()) Old news 00000800:00000200:8.2:1547251645.309265:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:12.0:1547251645.309280:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (20)++ 00000800:00000200:12.0:1547251645.309285:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (21)-- 00000800:00000200:12.0:1547251645.309286:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:12.0:1547251645.309287:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (19)-- 00000800:00000200:12.0:1547251645.309290:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07d800] (18)-- 00000800:00000200:8.2:1547251645.309421:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07d800] (17)++ 00000800:00000200:12.0:1547251645.309433:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (18)++ 00000800:00000200:12.0:1547251645.309437:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (19)-- 00000800:00000200:12.0:1547251645.309438:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (18)-- 00000800:00000200:12.0:1547251645.309439:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (17)++ 00000800:00000200:12.0:1547251645.309440:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (18)-- 00000800:00000200:12.0:1547251645.309440:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (17)-- 00000800:00000200:12.0:1547251645.309441:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (16)++ 00000800:00000200:12.0:1547251645.309442:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (17)-- 00000800:00000200:12.0:1547251645.309442:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (16)-- 00000800:00000200:12.0:1547251645.309444:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (15)++ 00000800:00000200:12.0:1547251645.309448:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (16)-- 00000800:00000200:12.0:1547251645.309448:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (15)-- 00000800:00000200:12.0:1547251645.309449:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (14)++ 00000800:00000200:12.0:1547251645.309451:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (15)-- 00000800:00000200:12.0:1547251645.309451:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (14)-- 00000800:00000200:15.0:1547251645.309453:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (13)++ 00000800:00000200:15.0:1547251645.309460:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (14)-- 00000800:00000200:15.0:1547251645.309461:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (13)-- 00000800:00000200:15.0:1547251645.309462:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (12)++ 00000800:00000200:15.0:1547251645.309465:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (13)-- 00000800:00000200:15.0:1547251645.309467:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (12)-- 00000800:00000200:12.0:1547251645.309468:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (11)++ 00000800:00000200:12.0:1547251645.309471:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (12)-- 00000800:00000200:12.0:1547251645.309472:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (11)-- 00000800:00000200:12.0:1547251645.309473:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (10)++ 00000800:00000200:15.0:1547251645.309474:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (11)++ 00000800:00000200:12.0:1547251645.309474:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (11)-- 00000800:00000200:12.0:1547251645.309475:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (11)-- 00000800:00000200:15.0:1547251645.309476:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (10)-- 00000800:00000200:12.0:1547251645.309477:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (10)++ 00000800:00000200:15.0:1547251645.309478:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (10)-- 00000800:00000200:15.0:1547251645.309479:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (9)++ 00000800:00000200:12.0:1547251645.309479:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (9)-- 00000800:00000200:12.0:1547251645.309480:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (9)-- 00000800:00000200:15.0:1547251645.309481:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (8)-- 00000800:00000200:12.0:1547251645.309481:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (8)++ 00000800:00000200:15.0:1547251645.309482:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (8)-- 00000800:00000200:15.0:1547251645.309483:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (7)++ 00000800:00000200:12.0:1547251645.309483:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (7)-- 00000800:00000200:12.0:1547251645.309485:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (7)-- 00000800:00000200:15.0:1547251645.309486:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (6)-- 00000800:00000200:12.0:1547251645.309486:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (6)++ 00000800:00000200:15.0:1547251645.309487:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (6)-- 00000800:00000200:15.0:1547251645.309488:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (5)++ 00000800:00000200:12.0:1547251645.309488:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (5)-- 00000800:00000200:12.0:1547251645.309489:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (5)-- 00000800:00000200:15.0:1547251645.309490:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (4)-- 00000800:00000200:12.0:1547251645.309490:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (4)++ 00000800:00000200:15.0:1547251645.309491:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (4)-- 00000800:00000200:15.0:1547251645.309492:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (3)-- 00000800:00000200:12.0:1547251645.309492:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (3)-- 00000800:00000200:12.0:1547251645.309493:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (1)-- 00000800:00000200:9.0:1547251645.310179:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (1)-- 00000100:00000200:9.0:1547251647.242715:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446480, portal 4 00000100:00000200:12.0:1547251647.242716:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446464, portal 4 00000100:00000200:9.0:1547251647.242719:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446480, offset 0 00000100:00000200:3.0:1547251647.242719:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446496, portal 4 00000100:00000200:12.0:1547251647.242720:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446464, offset 0 00000100:00000200:3.0:1547251647.242721:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446496, offset 0 00000400:00000200:9.0:1547251647.242722:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:12.0:1547251647.242725:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251647.242725:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:3.0:1547251647.242726:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251647.242727:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23066] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251647.242728:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:3.0:1547251647.242728:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:12.0:1547251647.242730:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23066] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251647.242730:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23064] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23066] 00000400:00000200:3.0:1547251647.242731:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23066] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251647.242732:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:12.0:1547251647.242734:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23064] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23066] 00000400:00000200:9.0:1547251647.242734:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251647.242734:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23065] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23066] 00000400:00000200:9.0:1547251647.242735:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251647.242736:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251647.242737:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9761, 9763] 00000400:00000200:3.0:1547251647.242737:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:12.0:1547251647.242738:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251647.242738:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251647.242739:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251647.242739:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:3.0:1547251647.242740:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251647.242741:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9762, 9763] 00000400:00000200:3.0:1547251647.242741:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9762, 9763] 00000400:00000200:12.0:1547251647.242743:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:9.0:1547251647.242744:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000400:00000200:3.0:1547251647.242744:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000800:00000200:9.0:1547251647.242746:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:12.0:1547251647.242750:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251647.242752:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:12.0:1547251647.242755:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000400:00000200:3.0:1547251647.242755:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251647.242757:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:12.0:1547251647.242758:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:3.0:1547251647.242758:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:9.0:1547251647.242759:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251647.242760:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:9.0:1547251647.242760:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:9.0:1547251647.242762:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:3.0:1547251647.242762:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251647.242763:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:9.0:1547251647.242764:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000800:00000200:3.0:1547251647.242764:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:3.0:1547251647.242767:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:7.2:1547251647.242809:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:7.2:1547251647.242819:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251647.242826:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:11.0:1547251647.242830:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (24)-- 00000100:00000200:11.0:1547251647.242835:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cae900 x1622384012446464/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251647.242842:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038ee0 00000800:00000200:10.0:1547251647.242842:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000400:00000200:11.0:1547251647.242844:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251647.242844:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000400:00000200:11.0:1547251647.242848:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000100:00000200:10.0:1547251647.242849:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf1500 x1622384012446480/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251647.242851:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:11.0:1547251647.242852:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:11.0:1547251647.242854:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000400:00000200:10.0:1547251647.242855:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaf68 00000400:00000200:10.0:1547251647.242856:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:11.0:1547251647.242857:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c2d00 x1622384012446496/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251647.242860:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251647.242862:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555bb0 00000800:00000200:10.0:1547251647.242862:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:11.0:1547251647.242863:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251647.242863:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:10.0:1547251647.242865:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:11.0:1547251647.242866:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251647.242868:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:7.2:1547251647.242932:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:10.0:1547251647.242951:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251647.242960:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251647.242965:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:10.0:1547251647.242965:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:5.2:1547251647.242969:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000400:00000200:10.0:1547251647.242970:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04b00 00000400:00000200:10.0:1547251647.242975:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd71f9 [1] + 192 00000800:00000200:5.2:1547251647.242978:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000100:00000200:10.0:1547251647.242979:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cae900 x1622384012446464/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251647.242980:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251647.242983:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251647.242987:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251647.242990:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:11.0:1547251647.242992:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04b10 00000800:00000200:10.0:1547251647.242992:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251647.242994:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:11.0:1547251647.242995:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd71e9 [1] + 192 00000800:00000200:10.0:1547251647.242995:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:10.0:1547251647.242997:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251647.242999:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000100:00000200:12.0:1547251647.243000:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cae900 x1622384012446464/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251647.243003:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000100:00000200:12.0:1547251647.243005:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cae900 x1622384012446464/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251647.243006:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04b20 00000400:00000200:12.0:1547251647.243009:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038dd0 00000100:00000200:11.0:1547251647.243012:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf1500 x1622384012446480/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:12.0:1547251647.243013:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cae900 x1622384012446464/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:11.0:1547251647.243022:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251647.243024:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000400:00000200:10.0:1547251647.243024:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7209 [1] + 192 00000800:00000200:11.0:1547251647.243026:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251647.243027:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000100:00000200:10.0:1547251647.243028:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c2d00 x1622384012446496/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251647.243029:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:11.0:1547251647.243030:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251647.243037:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251647.243039:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251647.243040:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251647.243041:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:9.0:1547251647.243042:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf1500 x1622384012446480/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251647.243048:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf1500 x1622384012446480/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251647.243052:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea660 00010000:00080000:9.0:1547251647.243056:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf1500 x1622384012446480/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251647.243091:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c2d00 x1622384012446496/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251647.243098:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c2d00 x1622384012446496/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251647.243102:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251647.243106:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c2d00 x1622384012446496/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251647.706685:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446512, portal 4 00000100:00000200:9.0:1547251647.706687:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446528, portal 4 00000100:00000200:3.0:1547251647.706688:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446512, offset 0 00000100:00000200:9.0:1547251647.706690:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446528, offset 0 00000400:00000200:3.0:1547251647.706691:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251647.706693:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:3.0:1547251647.706693:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251647.706696:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:3.0:1547251647.706696:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23066] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251647.706698:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23066] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251647.706699:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23067] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23066] 00000400:00000200:9.0:1547251647.706702:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23067] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23066] 00000400:00000200:3.0:1547251647.706702:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251647.706703:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251647.706704:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251647.706706:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251647.706706:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251647.706707:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251647.706707:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9755, 9756] 00000400:00000200:9.0:1547251647.706709:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9755, 9756] 00000400:00000200:3.0:1547251647.706709:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:9.0:1547251647.706710:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:3.0:1547251647.706714:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251647.706717:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:9.0:1547251647.706720:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251647.706720:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251647.706721:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:9.0:1547251647.706722:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:3.0:1547251647.706722:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:3.0:1547251647.706725:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:9.0:1547251647.706726:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251647.706727:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (21)++ 00000800:00000200:9.0:1547251647.706728:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (22)++ 00000800:00000200:9.0:1547251647.706731:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (23)-- 00000100:00000200:15.0:1547251647.706733:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446544, portal 4 00000100:00000200:15.0:1547251647.706737:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446544, offset 0 00000400:00000200:15.0:1547251647.706741:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:15.0:1547251647.706744:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:12.2:1547251647.706745:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (22)++ 00000400:00000200:15.0:1547251647.706747:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:254, d:10, s:23068] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251647.706750:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23067] with best_ni 10.0.11.90@o2ib10 [c:254, d:10, s:23068] 00000400:00000200:15.0:1547251647.706752:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:15.0:1547251647.706753:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251647.706755:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251647.706756:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[6, 8], s:[9757, 9756] 00000400:00000200:15.0:1547251647.706758:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:15.0:1547251647.706763:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251647.706765:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:15.0:1547251647.706769:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251647.706770:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:15.0:1547251647.706771:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:15.0:1547251647.706774:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:0.2:1547251647.706789:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251647.706801:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (23)++ 00000800:00000200:5.0:1547251647.706806:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (24)-- 00000100:00000200:5.0:1547251647.706811:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c2a00 x1622384012446512/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251647.706815:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000400:00000200:5.0:1547251647.706818:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245555d8 00000800:00000200:4.0:1547251647.706818:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000400:00000200:5.0:1547251647.706820:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:4.0:1547251647.706821:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f2100 x1622384012446544/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251647.706823:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251647.706827:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251647.706828:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000400:00000200:4.0:1547251647.706828:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1990 00000400:00000200:4.0:1547251647.706829:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:5.0:1547251647.706830:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000100:00000200:5.0:1547251647.706832:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf2d00 x1622384012446528/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251647.706832:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251647.706835:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251647.706837:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000400:00000200:5.0:1547251647.706838:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaf68 00000800:00000200:4.0:1547251647.706838:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:5.0:1547251647.706839:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251647.706842:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251647.706844:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:12.2:1547251647.706910:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:0.2:1547251647.706916:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251647.706921:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:4.0:1547251647.706924:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251647.706928:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:4.0:1547251647.706928:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.92@o2ib10 00000800:00000200:5.0:1547251647.706931:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251647.706933:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251647.706935:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251647.706938:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04b30 00000400:00000200:5.0:1547251647.706939:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04b40 00000400:00000200:4.0:1547251647.706951:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe40b5 [1] + 192 00000400:00000200:5.0:1547251647.706962:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe40c5 [1] + 192 00000100:00000200:4.0:1547251647.706966:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c2a00 x1622384012446512/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251647.706980:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251647.706981:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf2d00 x1622384012446528/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251647.706982:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251647.706984:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:4.0:1547251647.706985:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251647.706988:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251647.706990:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251647.706993:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251647.706994:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:5.0:1547251647.706995:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000100:00000200:3.0:1547251647.706995:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c2a00 x1622384012446512/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251647.706996:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251647.706998:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:4.0:1547251647.706998:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04b50 00000800:00000200:5.0:1547251647.706999:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:5.0:1547251647.707001:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:3.0:1547251647.707001:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c2a00 x1622384012446512/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251647.707002:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:3.0:1547251647.707005:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00000400:00000200:4.0:1547251647.707008:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe40d5 [1] + 192 00010000:00080000:3.0:1547251647.707008:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c2a00 x1622384012446512/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251647.707012:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f2100 x1622384012446544/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251647.707022:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251647.707024:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000100:00000200:9.0:1547251647.707025:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf2d00 x1622384012446528/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251647.707025:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251647.707026:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:9.0:1547251647.707030:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf2d00 x1622384012446528/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251647.707035:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea660 00000100:00000200:15.0:1547251647.707038:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f2100 x1622384012446544/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251647.707038:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf2d00 x1622384012446528/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:15.0:1547251647.707048:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f2100 x1622384012446544/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251647.707052:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb15d8 00010000:00080000:15.0:1547251647.707056:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f2100 x1622384012446544/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251654 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251648.106694:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446560, portal 4 00000100:00000200:9.0:1547251648.106697:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446560, offset 0 00000400:00000200:9.0:1547251648.106700:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251648.106703:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251648.106705:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04b60 00000400:00000200:9.0:1547251648.106708:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4a09 [1] + 0 00000100:00000200:4.0:1547251648.106708:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446576, portal 4 00000400:00000200:9.0:1547251648.106710:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ba9d367dd0 00000100:00000200:9.0:1547251648.106712:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:4.0:1547251648.106712:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446576, offset 0 00000400:00000200:4.0:1547251648.106715:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251648.106718:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251648.106720:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba9d367dd0 00000400:00000200:9.0:1547251648.106721:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251648.106721:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04b70 00000400:00000200:4.0:1547251648.106723:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2975 [1] + 0 00000100:00000200:9.0:1547251648.106725:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf3c00 x1622384012446560/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251655 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251648.106725:0:12676:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17ea770 00000100:00000200:4.0:1547251648.106727:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:4.0:1547251648.106735:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea770 00000400:00000200:4.0:1547251648.106736:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251648.106738:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea000 00000100:00000200:4.0:1547251648.106738:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c431e00 x1622384012446576/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251655 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251648.106739:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251648.106740:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251648.106743:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446560 00000400:00000200:4.0:1547251648.106743:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bcc0 00000400:00000200:4.0:1547251648.106744:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251648.106745:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00010000:00000200:8.0:1547251648.106752:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3d9850 x1622384012446560/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:709/0 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:8.0:1547251648.106761:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446560, offset 192 00000400:00000200:8.0:1547251648.106763:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251648.106766:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251648.106768:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04b60 00000100:00000200:7.0:1547251648.106768:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446576 00000400:00000200:8.0:1547251648.106770:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf241 [1] + 192 00000100:00000200:8.0:1547251648.106773:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf3c00 x1622384012446560/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251655 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00000200:7.0:1547251648.106777:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a99d398c50 x1622384012446576/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:709/0 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:8.0:1547251648.106783:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251648.106785:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688cc0 00000100:00000200:7.0:1547251648.106785:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446576, offset 192 00000400:00000200:8.0:1547251648.106786:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251648.106787:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251648.106788:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251648.106790:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:9.0:1547251648.106792:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf3c00 x1622384012446560/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251655 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251648.106793:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04b70 00000100:00000200:8.0:1547251648.106795:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:9.0:1547251648.106798:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf3c00 x1622384012446560/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251655 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251648.106803:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea660 00000400:00000200:7.0:1547251648.106805:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf251 [1] + 192 00010000:00080000:9.0:1547251648.106807:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf3c00 x1622384012446560/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251655 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251648.106809:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c431e00 x1622384012446576/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251655 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251648.106819:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251648.106821:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eab28 00000400:00000200:7.0:1547251648.106822:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251648.106823:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251648.106826:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c431e00 x1622384012446576/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251655 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251648.106828:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:4.0:1547251648.106836:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c431e00 x1622384012446576/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251655 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251648.106840:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bc38 00010000:00080000:4.0:1547251648.106843:0:12676:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae5c431e00 x1622384012446576/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251655 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251648.122684:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446592, portal 4 00000100:00000200:5.0:1547251648.122687:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446592, offset 0 00000400:00000200:5.0:1547251648.122689:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251648.122692:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251648.122695:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04b80 00000400:00000200:5.0:1547251648.122697:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe29a5 [1] + 0 00000400:00000200:5.0:1547251648.122700:0:12676:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17ea330 00000100:00000200:5.0:1547251648.122701:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:5.0:1547251648.122709:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea330 00000400:00000200:5.0:1547251648.122711:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251648.122713:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b82444f200 x1622384012446592/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251655 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251648.122718:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad5d8 00000400:00000200:5.0:1547251648.122718:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251648.122719:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251648.122739:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446592 00010000:00000200:7.0:1547251648.122748:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c2d8850 x1622384012446592/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:709/0 lens 224/368 e 0 to 0 dl 1547251654 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:7.0:1547251648.122755:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446592, offset 192 00000400:00000200:7.0:1547251648.122757:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251648.122760:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251648.122763:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04b80 00000400:00000200:7.0:1547251648.122766:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf261 [1] + 192 00000100:00000200:7.0:1547251648.122769:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b82444f200 x1622384012446592/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251655 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251648.122778:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251648.122780:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea990 00000400:00000200:7.0:1547251648.122781:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251648.122782:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251648.122786:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b82444f200 x1622384012446592/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251655 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251648.122787:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251648.122791:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b82444f200 x1622384012446592/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251655 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251648.122795:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad2a8 00010000:00080000:5.0:1547251648.122798:0:12676:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b82444f200 x1622384012446592/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251655 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251650.138682:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446608, portal 4 00000100:00000200:5.0:1547251650.138684:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446608, offset 0 00000400:00000200:5.0:1547251650.138687:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:5.0:1547251650.138691:0:12675:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:5.0:1547251650.138694:0:12675:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23068] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:5.0:1547251650.138697:0:12675:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23068] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23068] 00000400:00000200:5.0:1547251650.138699:0:12675:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:5.0:1547251650.138701:0:12675:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:5.0:1547251650.138702:0:12675:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:5.0:1547251650.138703:0:12675:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9756, 9759] 00000400:00000200:5.0:1547251650.138705:0:12675:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:5.0:1547251650.138710:0:12675:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:5.0:1547251650.138713:0:12675:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:5.0:1547251650.138716:0:12675:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:5.0:1547251650.138717:0:12675:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:5.0:1547251650.138718:0:12675:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:5.0:1547251650.138721:0:12675:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:7.2:1547251650.138737:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:0.0:1547251650.138748:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446624, portal 4 00000100:00000200:0.0:1547251650.138752:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446624, offset 0 00000400:00000200:0.0:1547251650.138756:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:0.0:1547251650.138760:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:0.0:1547251650.138763:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23069] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251650.138768:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23068] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23069] 00000400:00000200:0.0:1547251650.138770:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251650.138772:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251650.138773:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251650.138774:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9757, 9759] 00000400:00000200:0.0:1547251650.138776:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:0.0:1547251650.138781:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000100:00000200:10.0:1547251650.138784:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446640, portal 4 00000800:00000200:0.0:1547251650.138784:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000100:00000200:10.0:1547251650.138787:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446640, offset 0 00000800:00000200:0.0:1547251650.138787:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251650.138788:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:0.0:1547251650.138789:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251650.138791:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000400:00000200:10.0:1547251650.138791:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:0.0:1547251650.138792:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:10.0:1547251650.138794:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:10.0:1547251650.138796:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23069] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:10.0:1547251650.138799:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23069] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23069] 00000800:00000200:15.0:1547251650.138800:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000400:00000200:10.0:1547251650.138802:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:10.0:1547251650.138803:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.2:1547251650.138803:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000100:00000200:15.0:1547251650.138804:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824449800 x1622384012446608/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251657 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251650.138804:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251650.138806:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9757, 9760] 00000400:00000200:10.0:1547251650.138807:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251650.138809:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad5d8 00000400:00000200:15.0:1547251650.138811:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251650.138812:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:15.0:1547251650.138813:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251650.138815:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:10.0:1547251650.138815:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:15.0:1547251650.138817:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:12.0:1547251650.138818:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:10.0:1547251650.138818:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:10.0:1547251650.138819:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:10.0:1547251650.138820:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251650.138823:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:10.0:1547251650.138823:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:15.0:1547251650.138825:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:12.0:1547251650.138826:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f4b00 x1622384012446624/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251657 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251650.138832:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665000 00000400:00000200:12.0:1547251650.138833:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:7.2:1547251650.138834:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:12.0:1547251650.138836:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251650.138839:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:12.0:1547251650.138840:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251650.138842:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:12.0:1547251650.138844:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960e8600 x1622384012446640/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251657 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251650.138846:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:12.0:1547251650.138848:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a9146e8 00000400:00000200:12.0:1547251650.138849:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251650.138851:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251650.138852:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:7.2:1547251650.138896:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:9.2:1547251650.138901:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:15.0:1547251650.138907:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:15.0:1547251650.138910:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251650.138913:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251650.138914:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251650.138916:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251650.138917:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04b90 00000800:00000200:9.2:1547251650.138917:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000400:00000200:15.0:1547251650.138920:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fe1d [1] + 192 00000400:00000200:12.0:1547251650.138921:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000100:00000200:15.0:1547251650.138924:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824449800 x1622384012446608/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251657 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251650.138925:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04bb0 00000400:00000200:15.0:1547251650.138933:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251650.138935:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000400:00000200:12.0:1547251650.138935:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fe3d [1] + 192 00000800:00000200:15.0:1547251650.138937:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:15.0:1547251650.138938:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:12.0:1547251650.138938:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960e8600 x1622384012446640/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251657 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251650.138939:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:15.0:1547251650.138940:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:15.0:1547251650.138949:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251650.138951:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000400:00000200:15.0:1547251650.138954:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251650.138955:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:15.0:1547251650.138956:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04ba0 00000800:00000200:12.0:1547251650.138957:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251650.138959:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:5.0:1547251650.138959:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824449800 x1622384012446608/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251657 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251650.138960:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:12.0:1547251650.138961:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:5.0:1547251650.138964:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824449800 x1622384012446608/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251657 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251650.138968:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad2a8 00000400:00000200:15.0:1547251650.138971:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fe2d [1] + 192 00010000:00080000:5.0:1547251650.138972:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824449800 x1622384012446608/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251657 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251650.138975:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960e8600 x1622384012446640/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251657 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:10.0:1547251650.138980:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960e8600 x1622384012446640/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251657 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251650.138984:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914dd0 00000100:00000200:15.0:1547251650.138986:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f4b00 x1622384012446624/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251657 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:10.0:1547251650.138988:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba960e8600 x1622384012446640/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251657 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:15.0:1547251650.138994:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251650.138996:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251650.138997:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:15.0:1547251650.138998:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:0.0:1547251650.139004:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f4b00 x1622384012446624/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251657 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251650.139009:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f4b00 x1622384012446624/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251657 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251650.139018:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00010000:00080000:0.0:1547251650.139022:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f4b00 x1622384012446624/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251657 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251652.250686:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446656, portal 4 00000100:00000200:8.0:1547251652.250688:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446656, offset 0 00000400:00000200:8.0:1547251652.250692:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251652.250694:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251652.250697:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23070] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251652.250700:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23069] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23070] 00000400:00000200:8.0:1547251652.250702:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:8.0:1547251652.250704:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251652.250705:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251652.250706:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9762, 9765] 00000400:00000200:8.0:1547251652.250708:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:8.0:1547251652.250713:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251652.250715:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:8.0:1547251652.250718:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251652.250719:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:8.0:1547251652.250720:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:8.0:1547251652.250723:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000100:00000200:14.0:1547251652.250737:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446672, portal 4 00000100:00000200:14.0:1547251652.250741:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446672, offset 0 00000400:00000200:14.0:1547251652.250745:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:14.0:1547251652.250749:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:14.0:1547251652.250751:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23070] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:14.0:1547251652.250755:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23070] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23070] 00000400:00000200:14.0:1547251652.250757:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:14.0:1547251652.250758:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251652.250759:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:0.0:1547251652.250760:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446688, portal 4 00000400:00000200:14.0:1547251652.250761:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9763, 9765] 00000400:00000200:14.0:1547251652.250763:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000100:00000200:0.0:1547251652.250763:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446688, offset 0 00000400:00000200:14.0:1547251652.250767:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000400:00000200:0.0:1547251652.250768:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:14.0:1547251652.250770:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:0.0:1547251652.250771:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:14.0:1547251652.250773:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:7.2:1547251652.250773:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000400:00000200:0.0:1547251652.250774:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23071] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:14.0:1547251652.250775:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:14.0:1547251652.250776:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000400:00000200:0.0:1547251652.250777:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23070] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23071] 00000800:00000200:14.0:1547251652.250779:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000400:00000200:0.0:1547251652.250779:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251652.250781:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251652.250782:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251652.250783:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9763, 9766] 00000400:00000200:0.0:1547251652.250785:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:0.0:1547251652.250790:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251652.250792:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:0.0:1547251652.250795:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251652.250797:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:0.0:1547251652.250797:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (22)++ 00000800:00000200:0.0:1547251652.250798:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (23)++ 00000800:00000200:0.0:1547251652.250801:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (25)-- 00000800:00000200:11.0:1547251652.250807:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000100:00000200:11.0:1547251652.250812:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824752a00 x1622384012446656/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251652.250814:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:10.0:1547251652.250816:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000400:00000200:11.0:1547251652.250819:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688f68 00000100:00000200:10.0:1547251652.250819:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f4800 x1622384012446688/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251652.250821:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251652.250825:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251652.250826:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665dd0 00000400:00000200:10.0:1547251652.250827:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:5.2:1547251652.250828:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251652.250830:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000400:00000200:10.0:1547251652.250830:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251652.250832:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:11.0:1547251652.250833:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:10.0:1547251652.250834:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251652.250838:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251652.250840:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000100:00000200:10.0:1547251652.250840:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af6300 x1622384012446672/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251652.250846:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:10.0:1547251652.250847:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251652.250850:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251652.250852:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:7.2:1547251652.250908:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:5.2:1547251652.250914:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251652.250920:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:11.0:1547251652.250923:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251652.250927:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.1.11.91@o2ib10 00000800:00000200:10.0:1547251652.250928:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251652.250930:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251652.250932:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251652.250935:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251652.250937:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04bc0 00000400:00000200:10.0:1547251652.250938:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04be0 00000400:00000200:11.0:1547251652.250941:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7279 [1] + 192 00000400:00000200:10.0:1547251652.250955:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd72a9 [1] + 192 00000100:00000200:11.0:1547251652.250960:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824752a00 x1622384012446656/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251652.250970:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251652.250972:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000100:00000200:10.0:1547251652.250972:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f4800 x1622384012446688/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251652.250974:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251652.250976:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:11.0:1547251652.250978:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251652.250980:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251652.250982:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:11.0:1547251652.250984:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251652.250984:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000100:00000200:8.0:1547251652.250984:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824752a00 x1622384012446656/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251652.250986:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251652.250987:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:11.0:1547251652.250988:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04bd0 00000800:00000200:10.0:1547251652.250989:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:10.0:1547251652.250990:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:8.0:1547251652.250990:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824752a00 x1622384012446656/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251652.250991:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:8.0:1547251652.250994:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688dd0 00000400:00000200:11.0:1547251652.250998:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7299 [1] + 192 00010000:00080000:8.0:1547251652.250998:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824752a00 x1622384012446656/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:0.0:1547251652.251003:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f4800 x1622384012446688/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251652.251009:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f4800 x1622384012446688/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251652.251013:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00000100:00000200:11.0:1547251652.251015:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af6300 x1622384012446672/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251652.251016:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f4800 x1622384012446688/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:11.0:1547251652.251025:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251652.251026:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251652.251028:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251652.251029:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:14.0:1547251652.251036:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af6300 x1622384012446672/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:14.0:1547251652.251041:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af6300 x1622384012446672/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251652.251046:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00010000:00080000:14.0:1547251652.251049:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af6300 x1622384012446672/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251652.714684:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446704, portal 4 00000100:00000200:8.0:1547251652.714688:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446704, offset 0 00000400:00000200:8.0:1547251652.714691:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251652.714694:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251652.714696:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23071] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251652.714699:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23071] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23071] 00000400:00000200:8.0:1547251652.714702:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251652.714703:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251652.714704:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251652.714705:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9757, 9757] 00000400:00000200:8.0:1547251652.714707:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:8.0:1547251652.714712:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251652.714714:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:8.0:1547251652.714717:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251652.714719:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:8.0:1547251652.714720:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:8.0:1547251652.714722:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000100:00000200:1.0:1547251652.714733:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446720, portal 4 00000100:00000200:1.0:1547251652.714737:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446720, offset 0 00000400:00000200:1.0:1547251652.714741:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:1.0:1547251652.714744:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:12.2:1547251652.714746:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000400:00000200:1.0:1547251652.714747:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23072] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251652.714750:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23071] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23072] 00000400:00000200:1.0:1547251652.714752:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:1.0:1547251652.714754:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251652.714755:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:14.0:1547251652.714756:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446736, portal 4 00000400:00000200:1.0:1547251652.714756:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9757, 9758] 00000400:00000200:1.0:1547251652.714758:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000100:00000200:14.0:1547251652.714759:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446736, offset 0 00000400:00000200:1.0:1547251652.714763:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:14.0:1547251652.714764:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:1.0:1547251652.714765:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:14.0:1547251652.714766:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:1.0:1547251652.714768:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:14.0:1547251652.714769:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23072] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:1.0:1547251652.714770:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:1.0:1547251652.714771:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:14.0:1547251652.714772:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23072] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23072] 00000400:00000200:14.0:1547251652.714774:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:1.0:1547251652.714774:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:14.0:1547251652.714776:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251652.714777:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251652.714778:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9758, 9758] 00000400:00000200:14.0:1547251652.714780:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:14.0:1547251652.714785:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251652.714788:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:14.0:1547251652.714791:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251652.714792:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (22)++ 00000800:00000200:0.2:1547251652.714792:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:14.0:1547251652.714793:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00000800:00000200:14.0:1547251652.714796:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (24)-- 00000800:00000200:5.0:1547251652.714802:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:5.0:1547251652.714806:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000100:00000200:5.0:1547251652.714812:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824756600 x1622384012446704/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251652.714819:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000400:00000200:5.0:1547251652.714820:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688110 00000400:00000200:5.0:1547251652.714821:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251652.714821:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:5.0:1547251652.714825:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000100:00000200:4.0:1547251652.714825:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f4500 x1622384012446720/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251652.714828:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251652.714830:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251652.714831:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000400:00000200:4.0:1547251652.714831:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2660 00000400:00000200:4.0:1547251652.714833:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:5.0:1547251652.714835:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f2400 x1622384012446736/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251652.714836:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251652.714838:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251652.714840:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000400:00000200:5.0:1547251652.714841:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:5.0:1547251652.714842:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251652.714842:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:5.0:1547251652.714845:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251652.714847:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:0.2:1547251652.714898:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:12.2:1547251652.714903:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:4.0:1547251652.714911:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251652.714915:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000800:00000200:5.0:1547251652.714916:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251652.714918:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000800:00000200:0.2:1547251652.714919:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000400:00000200:4.0:1547251652.714920:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251652.714923:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251652.714925:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04c10 00000400:00000200:5.0:1547251652.714927:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04bf0 00000400:00000200:4.0:1547251652.714929:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4175 [1] + 192 00000400:00000200:5.0:1547251652.714934:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4155 [1] + 192 00000100:00000200:4.0:1547251652.714938:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f2400 x1622384012446736/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251652.714959:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251652.714961:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824756600 x1622384012446704/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251652.714962:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251652.714964:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:4.0:1547251652.714965:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251652.714967:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251652.714969:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251652.714970:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:5.0:1547251652.714971:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251652.714973:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251652.714973:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000100:00000200:14.0:1547251652.714974:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f2400 x1622384012446736/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251652.714975:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251652.714976:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000400:00000200:4.0:1547251652.714977:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:5.0:1547251652.714978:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:14.0:1547251652.714979:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f2400 x1622384012446736/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251652.714981:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04c00 00000400:00000200:14.0:1547251652.714983:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00000100:00000200:8.0:1547251652.714986:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824756600 x1622384012446704/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:14.0:1547251652.714987:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f2400 x1622384012446736/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251652.714992:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824756600 x1622384012446704/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251652.714996:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688dd0 00000400:00000200:4.0:1547251652.714999:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4165 [1] + 192 00010000:00080000:8.0:1547251652.715000:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824756600 x1622384012446704/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251652.715003:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f4500 x1622384012446720/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251652.715011:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251652.715012:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:4.0:1547251652.715013:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251652.715014:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:1.0:1547251652.715021:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f4500 x1622384012446720/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:1.0:1547251652.715027:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f4500 x1622384012446720/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251652.715031:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00010000:00080000:1.0:1547251652.715034:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f4500 x1622384012446720/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251659 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251653.114687:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446752, portal 4 00000100:00000200:8.0:1547251653.114690:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446752, offset 0 00000400:00000200:8.0:1547251653.114693:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251653.114695:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251653.114698:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04c20 00000400:00000200:8.0:1547251653.114700:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4ae9 [1] + 0 00000400:00000200:8.0:1547251653.114702:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d688908 00000100:00000200:5.0:1547251653.114702:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446768, portal 4 00000100:00000200:8.0:1547251653.114704:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:5.0:1547251653.114705:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446768, offset 0 00000400:00000200:5.0:1547251653.114708:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251653.114710:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251653.114713:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688908 00000400:00000200:5.0:1547251653.114713:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04c30 00000400:00000200:8.0:1547251653.114715:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251653.114716:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2a15 [1] + 0 00000400:00000200:5.0:1547251653.114718:0:12675:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17eac38 00000100:00000200:8.0:1547251653.114719:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824751200 x1622384012446752/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251660 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251653.114720:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251653.114725:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688110 00000400:00000200:8.0:1547251653.114726:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251653.114728:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:5.0:1547251653.114729:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eac38 00000400:00000200:5.0:1547251653.114730:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251653.114732:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824448300 x1622384012446768/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251660 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251653.114736:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446752 00000400:00000200:5.0:1547251653.114737:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adbb0 00000400:00000200:5.0:1547251653.114738:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251653.114740:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00010000:00000200:9.0:1547251653.114745:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3d9450 x1622384012446752/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:714/0 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:9.0:1547251653.114753:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446752, offset 192 00000400:00000200:9.0:1547251653.114755:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251653.114758:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251653.114760:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04c20 00000400:00000200:9.0:1547251653.114762:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf2a1 [1] + 192 00000100:00000200:6.0:1547251653.114762:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446768 00000100:00000200:9.0:1547251653.114765:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824751200 x1622384012446752/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251660 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00000200:6.0:1547251653.114771:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c2d8450 x1622384012446768/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:714/0 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:9.0:1547251653.114773:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251653.114775:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea660 00000400:00000200:9.0:1547251653.114776:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:6.0:1547251653.114777:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446768, offset 192 00000400:00000200:9.0:1547251653.114779:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251653.114781:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:8.0:1547251653.114783:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824751200 x1622384012446752/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251660 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251653.114784:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:9.0:1547251653.114785:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:6.0:1547251653.114787:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04c30 00000100:00000200:8.0:1547251653.114789:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824751200 x1622384012446752/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251660 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251653.114794:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688dd0 00000400:00000200:6.0:1547251653.114797:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf2b1 [1] + 192 00010000:00080000:8.0:1547251653.114798:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824751200 x1622384012446752/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251660 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251653.114800:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824448300 x1622384012446768/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251660 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251653.114809:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251653.114811:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990220 00000400:00000200:6.0:1547251653.114812:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251653.114813:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251653.114817:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824448300 x1622384012446768/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251660 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251653.114819:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251653.114822:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824448300 x1622384012446768/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251660 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251653.114826:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad2a8 00010000:00080000:5.0:1547251653.114829:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824448300 x1622384012446768/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251660 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251653.130731:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446784, portal 4 00000100:00000200:7.0:1547251653.130735:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446784, offset 0 00000400:00000200:7.0:1547251653.130738:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251653.130741:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251653.130744:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04c40 00000400:00000200:7.0:1547251653.130747:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2a45 [1] + 0 00000400:00000200:7.0:1547251653.130749:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17ea198 00000100:00000200:7.0:1547251653.130751:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:7.0:1547251653.130760:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea198 00000400:00000200:7.0:1547251653.130761:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251653.130765:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c430f00 x1622384012446784/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251660 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251653.130771:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea330 00000400:00000200:7.0:1547251653.130772:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251653.130774:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251653.130784:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446784 00010000:00000200:6.0:1547251653.130794:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c2d8050 x1622384012446784/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:714/0 lens 224/368 e 0 to 0 dl 1547251659 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251653.130801:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012446784, offset 192 00000400:00000200:6.0:1547251653.130803:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251653.130806:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251653.130808:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04c40 00000400:00000200:6.0:1547251653.130811:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf2c1 [1] + 192 00000100:00000200:6.0:1547251653.130813:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c430f00 x1622384012446784/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251660 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251653.130822:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251653.130825:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990000 00000400:00000200:6.0:1547251653.130826:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251653.130828:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251653.130833:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c430f00 x1622384012446784/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251660 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251653.130835:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251653.130839:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c430f00 x1622384012446784/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251660 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251653.130844:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eac38 00010000:00080000:7.0:1547251653.130848:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae5c430f00 x1622384012446784/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251660 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:0.2:1547251655.005384:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251655.005401:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:4.0:1547251655.005442:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251655.005446:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251655.005447:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000800:00000200:5.0:1547251655.005450:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251655.005453:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251655.005454:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251655.005458:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b10320 00000400:00000200:4.0:1547251655.005458:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b10330 00000400:00000200:5.0:1547251655.005462:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 62944 00000400:00000200:4.0:1547251655.005466:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 63168 00000100:00000200:5.0:1547251655.005469:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:5.0:1547251655.005479:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251655.005479:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:5.0:1547251655.005481:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251655.005484:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251655.005485:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:4.0:1547251655.005486:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251655.005487:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251655.005488:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)++ 00000800:00000200:5.0:1547251655.005489:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251655.005490:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:4.0:1547251655.005491:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:5.0:1547251655.005493:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251655.005493:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251655.005495:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251655.005497:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 26 MB=0x5c38cb6b10300 00000400:00000200:4.0:1547251655.005499:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251655.005501:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0xd52b9 [1] + 0 00000400:00000200:4.0:1547251655.005503:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b10310 00000400:00000200:5.0:1547251655.005504:0:12662:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b17b038d48 00000100:00000200:5.0:1547251655.005506:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:4.0:1547251655.005506:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 63392 00000100:00000200:4.0:1547251655.005508:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:5.0:1547251655.005512:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038d48 00000400:00000200:4.0:1547251655.005512:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251655.005514:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251655.005514:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251655.005515:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251655.005515:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251655.005516:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251655.005517:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251655.005518:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251655.005519:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:5.0:1547251655.005520:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:4.0:1547251655.005542:0:12981:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011445024 00000100:00000200:7.0:1547251655.005545:0:12979:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011445008 00000100:00000200:6.0:1547251655.005547:0:13338:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011445040 00010000:00000200:4.0:1547251655.005549:0:12981:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae5c435100 x1622384011445024/t0(0) o400->cache1-MDT0000-lwp-OST0006_UUID@10.0.11.92@o2ib10:716/0 lens 224/224 e 0 to 0 dl 1547251661 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:7.0:1547251655.005554:0:12979:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b824448000 x1622384011445008/t0(0) o400->cache1-MDT0000-lwp-OST0007_UUID@10.0.11.92@o2ib10:716/0 lens 224/224 e 0 to 0 dl 1547251661 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:6.0:1547251655.005554:0:13338:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444f500 x1622384011445040/t0(0) o400->cache1-MDT0000-lwp-OST0008_UUID@10.0.11.92@o2ib10:716/0 lens 224/224 e 0 to 0 dl 1547251661 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:4.0:1547251655.005558:0:12981:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011445024, offset 192 00000400:00000200:4.0:1547251655.005564:0:12981:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000100:00000200:6.0:1547251655.005566:0:13338:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011445040, offset 192 00000100:00000200:7.0:1547251655.005567:0:12979:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011445008, offset 192 00000400:00000200:4.0:1547251655.005568:0:12981:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:7.0:1547251655.005570:0:12979:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:4.0:1547251655.005571:0:12981:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:2.0:1547251655.005572:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011444992 00000400:00000200:6.0:1547251655.005573:0:13338:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000400:00000200:4.0:1547251655.005573:0:12981:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251655.005574:0:12981:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9758, 9759] 00000400:00000200:4.0:1547251655.005576:0:12981:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00010000:00000200:2.0:1547251655.005578:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692fec050 x1622384011444992/t0(0) o400->04d65b15-cf9c-7ded-611d-ffe7fe3f3abf@10.0.11.92@o2ib10:716/0 lens 224/224 e 0 to 0 dl 1547251661 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:4.0:1547251655.005581:0:12981:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000100:00000200:2.0:1547251655.005596:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384011444992, offset 192 00000400:00000200:2.0:1547251655.005599:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000800:00000200:4.0:1547251655.005602:0:12981:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:2.0:1547251655.005602:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00000400:00000200:2.0:1547251655.005605:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251655.005606:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251655.005607:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9759, 9759] 00000400:00000200:2.0:1547251655.005609:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251655.005632:0:12979:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:7.0:1547251655.005635:0:12979:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251655.005637:0:12979:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251655.005639:0:12979:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9759, 9760] 00000400:00000200:7.0:1547251655.005641:0:12979:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251655.005648:0:12979:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:7.0:1547251655.005652:0:12979:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:6.0:1547251655.005653:0:13338:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00000400:00000200:6.0:1547251655.005656:0:13338:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251655.005659:0:13338:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251655.005660:0:13338:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9759, 9761] 00000400:00000200:6.0:1547251655.005663:0:13338:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:6.0:1547251655.005669:0:13338:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.1.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:6.0:1547251655.005673:0:13338:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:4.0:1547251655.005674:0:12981:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:4.0:1547251655.005675:0:12981:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:4.0:1547251655.005676:0:12981:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000400:00000200:2.0:1547251655.005676:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.1.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:4.0:1547251655.005679:0:12981:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:2.0:1547251655.005679:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:7.0:1547251655.005681:0:12979:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:6.0:1547251655.005681:0:13338:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:7.0:1547251655.005683:0:12979:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:6.0:1547251655.005683:0:13338:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:2.0:1547251655.005683:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:6.0:1547251655.005684:0:13338:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:2.0:1547251655.005684:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:7.0:1547251655.005685:0:12979:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:2.0:1547251655.005685:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:7.0:1547251655.005688:0:12979:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:6.0:1547251655.005688:0:13338:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:2.0:1547251655.005688:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:12.2:1547251655.005691:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000100:00000200:2.0:1547251655.005692:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:12.2:1547251655.005696:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:0.2:1547251655.005701:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:0.2:1547251655.005703:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251655.005706:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251655.005708:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000400:00000200:4.0:1547251655.005710:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bc38 00000400:00000200:4.0:1547251655.005712:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:5.0:1547251655.005713:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000400:00000200:4.0:1547251655.005715:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251655.005716:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000400:00000200:5.0:1547251655.005718:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eac38 00000800:00000200:4.0:1547251655.005718:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:5.0:1547251655.005719:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251655.005719:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251655.005721:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:5.0:1547251655.005722:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251655.005723:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990550 00000800:00000200:5.0:1547251655.005724:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:4.0:1547251655.005724:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:5.0:1547251655.005726:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000400:00000200:4.0:1547251655.005727:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251655.005728:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:4.0:1547251655.005729:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:5.0:1547251655.005730:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a6e8 00000800:00000200:4.0:1547251655.005730:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000400:00000200:5.0:1547251655.005731:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251655.005732:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:4.0:1547251655.005733:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000400:00000200:5.0:1547251655.005734:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251655.005735:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251655.005736:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:7.2:1547251655.039710:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251655.039723:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:12.0:1547251655.039766:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251655.039770:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251655.039773:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251655.039777:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510320 00000400:00000200:12.0:1547251655.039780:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 30368 00000800:00000200:15.0:1547251655.039781:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000100:00000200:12.0:1547251655.039783:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:15.0:1547251655.039785:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251655.039788:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251655.039792:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510310 00000400:00000200:12.0:1547251655.039793:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251655.039795:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000400:00000200:15.0:1547251655.039796:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 30592 00000100:00000200:15.0:1547251655.039799:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:12.0:1547251655.039799:0:12665:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (23)++ 00000800:00000200:12.0:1547251655.039802:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (24)-- 00000800:00000200:12.0:1547251655.039803:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (23)-- 00000400:00000200:15.0:1547251655.039804:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251655.039805:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:15.0:1547251655.039806:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251655.039807:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000800:00000200:15.0:1547251655.039809:0:12666:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (23)++ 00000800:00000200:15.0:1547251655.039811:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (24)-- 00000400:00000200:12.0:1547251655.039811:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:15.0:1547251655.039812:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:15.0:1547251655.039813:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:15.0:1547251655.039815:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251655.039815:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 26 MB=0x5c38cb3510300 00000400:00000200:15.0:1547251655.039818:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251655.039819:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0xd52c9 [1] + 0 00000400:00000200:15.0:1547251655.039820:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510330 00000400:00000200:12.0:1547251655.039821:0:12665:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b17b038b28 00000400:00000200:15.0:1547251655.039822:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 30816 00000100:00000200:15.0:1547251655.039824:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:12.0:1547251655.039824:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:12.0:1547251655.039831:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038b28 00000400:00000200:12.0:1547251655.039832:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:15.0:1547251655.039834:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251655.039834:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (23)++ 00000800:00000200:15.0:1547251655.039836:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)++ 00000800:00000200:12.0:1547251655.039836:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (24)-- 00000800:00000200:15.0:1547251655.039837:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (24)-- 00000800:00000200:12.0:1547251655.039838:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (23)-- 00000800:00000200:15.0:1547251655.039840:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (23)-- 00000100:00000200:13.0:1547251655.039841:0:13337:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821920 00000800:00000200:12.0:1547251655.039842:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:15.0:1547251655.039843:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251655.039844:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000800:00000200:15.0:1547251655.039845:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251655.039845:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:15.0:1547251655.039846:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00010000:00000200:13.0:1547251655.039847:0:13337:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97bae1af5a00 x1622383954821920/t0(0) o400->cache1-MDT0000-lwp-OST0009_UUID@10.0.11.93@o2ib10:716/0 lens 224/224 e 0 to 0 dl 1547251661 ref 1 fl Interpret:H/0/0 rc 0/0 00000800:00000200:12.0:1547251655.039848:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:15.0:1547251655.039849:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000100:00000200:14.0:1547251655.039855:0:12987:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821904 00000100:00000200:12.0:1547251655.039855:0:13371:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821936 00000100:00000200:13.0:1547251655.039856:0:13337:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821920, offset 192 00010000:00000200:12.0:1547251655.039859:0:13371:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b81d4a1e00 x1622383954821936/t0(0) o400->cache1-MDT0000-lwp-OST000b_UUID@10.0.11.93@o2ib10:716/0 lens 224/224 e 0 to 0 dl 1547251661 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:13.0:1547251655.039860:0:13337:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00010000:00000200:14.0:1547251655.039862:0:12987:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b81d4a1b00 x1622383954821904/t0(0) o400->cache1-MDT0000-lwp-OST000a_UUID@10.0.11.93@o2ib10:716/0 lens 224/224 e 0 to 0 dl 1547251661 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:13.0:1547251655.039864:0:13337:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000100:00000200:12.0:1547251655.039866:0:13371:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821936, offset 192 00000400:00000200:13.0:1547251655.039867:0:13337:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251655.039869:0:13337:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251655.039869:0:13371:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000100:00000200:14.0:1547251655.039871:0:12987:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821904, offset 192 00000400:00000200:13.0:1547251655.039871:0:13337:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9758, 9760] 00000400:00000200:13.0:1547251655.039873:0:13337:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251655.039875:0:12987:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000400:00000200:13.0:1547251655.039880:0:13337:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251655.039887:0:13337:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:12.0:1547251655.039888:0:13371:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00000100:00000200:2.0:1547251655.039889:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821888 00000400:00000200:12.0:1547251655.039890:0:13371:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251655.039892:0:13371:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251655.039894:0:13371:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9759, 9760] 00010000:00000200:2.0:1547251655.039895:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692fec450 x1622383954821888/t0(0) o400->b99b3dbb-ff3e-ba67-561d-3f0ab43da035@10.0.11.93@o2ib10:716/0 lens 224/224 e 0 to 0 dl 1547251661 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:12.0:1547251655.039896:0:13371:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000100:00000200:2.0:1547251655.039902:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954821888, offset 192 00000400:00000200:12.0:1547251655.039903:0:13371:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.1.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000400:00000200:2.0:1547251655.039905:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:12.0:1547251655.039906:0:13371:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:14.0:1547251655.039907:0:12987:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00000400:00000200:2.0:1547251655.039908:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:14.0:1547251655.039910:0:12987:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251655.039910:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251655.039912:0:12987:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251655.039912:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251655.039914:0:12987:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9759, 9761] 00000400:00000200:2.0:1547251655.039914:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9759, 9761] 00000400:00000200:2.0:1547251655.039915:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251655.039916:0:12987:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:14.0:1547251655.039921:0:12987:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.1.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251655.039924:0:12987:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:13.0:1547251655.039925:0:13337:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251655.039925:0:13371:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251655.039927:0:13337:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:12.0:1547251655.039927:0:13371:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:13.0:1547251655.039928:0:13337:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251655.039928:0:13371:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:2.0:1547251655.039928:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251655.039931:0:12987:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251655.039931:0:13337:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251655.039931:0:13371:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:2.0:1547251655.039931:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:14.0:1547251655.039933:0:12987:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:14.0:1547251655.039934:0:12987:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:2.0:1547251655.039934:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251655.039935:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:14.0:1547251655.039936:0:12987:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:2.0:1547251655.039936:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:2.0:1547251655.039939:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:7.2:1547251655.039951:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (22)++ 00000100:00000200:2.0:1547251655.039951:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:9.2:1547251655.039965:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:9.2:1547251655.039971:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:12.0:1547251655.039976:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)++ 00000800:00000200:12.0:1547251655.039978:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (24)-- 00000800:00000200:15.0:1547251655.039980:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000400:00000200:12.0:1547251655.039980:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b2a8 00000400:00000200:12.0:1547251655.039981:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:15.0:1547251655.039982:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000400:00000200:15.0:1547251655.039983:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038b28 00000400:00000200:12.0:1547251655.039983:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:15.0:1547251655.039984:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:12.0:1547251655.039985:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)-- 00000400:00000200:15.0:1547251655.039987:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251655.039987:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251655.039988:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:15.0:1547251655.039989:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:12.0:1547251655.039989:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00000800:00000200:15.0:1547251655.039990:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000400:00000200:12.0:1547251655.039990:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:15.0:1547251655.039992:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000400:00000200:12.0:1547251655.039992:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:15.0:1547251655.039994:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a4c8 00000800:00000200:12.0:1547251655.039994:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:15.0:1547251655.039995:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:12.0:1547251655.039996:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000400:00000200:15.0:1547251655.039997:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251655.039997:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:15.0:1547251655.039998:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251655.039999:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:5.2:1547251655.071492:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251655.071507:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:11.0:1547251655.071550:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251655.071554:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251655.071556:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251655.071558:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251655.071562:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251655.071562:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251655.071567:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510340 00000400:00000200:10.0:1547251655.071567:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 26 MB=0x5c38cb3510330 00000400:00000200:11.0:1547251655.071571:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 14112 00000400:00000200:10.0:1547251655.071577:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0xd5349 [1] + 0 00000400:00000200:10.0:1547251655.071579:0:12664:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b8246656e8 00000100:00000200:11.0:1547251655.071581:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:11.0:1547251655.071627:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251655.071628:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000800:00000200:11.0:1547251655.071630:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251655.071632:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251655.071633:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251655.071635:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000400:00000200:10.0:1547251655.071635:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246656e8 00000400:00000200:10.0:1547251655.071636:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251655.071638:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251655.071638:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251655.071640:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000400:00000200:11.0:1547251655.071642:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251655.071642:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251655.071644:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000400:00000200:11.0:1547251655.071646:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510360 00000800:00000200:10.0:1547251655.071646:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251655.071649:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 14336 00000400:00000200:10.0:1547251655.071651:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000100:00000200:11.0:1547251655.071652:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:10.0:1547251655.071654:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510350 00000400:00000200:11.0:1547251655.071656:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251655.071657:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000400:00000200:10.0:1547251655.071657:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 14560 00000800:00000200:11.0:1547251655.071659:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251655.071660:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000100:00000200:10.0:1547251655.071660:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:11.0:1547251655.071662:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:11.0:1547251655.071663:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:10.0:1547251655.071667:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251655.071668:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251655.071669:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251655.071670:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:11.0:1547251655.071684:0:12983:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821952 00000100:00000200:8.0:1547251655.071689:0:12982:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821984 00000100:00000200:10.0:1547251655.071690:0:13336:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821968 00010000:00000200:11.0:1547251655.071691:0:12983:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ba960eb600 x1622383954821952/t0(0) o400->cache1-MDT0000-lwp-OST0005_UUID@10.0.11.91@o2ib10:716/0 lens 224/224 e 0 to 0 dl 1547251661 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:10.0:1547251655.071694:0:13336:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b17b07b900 x1622383954821968/t0(0) o400->cache1-MDT0000-lwp-OST0003_UUID@10.0.11.91@o2ib10:716/0 lens 224/224 e 0 to 0 dl 1547251661 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:8.0:1547251655.071697:0:12982:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ba960ee600 x1622383954821984/t0(0) o400->cache1-MDT0000-lwp-OST0004_UUID@10.0.11.91@o2ib10:716/0 lens 224/224 e 0 to 0 dl 1547251661 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:11.0:1547251655.071699:0:12983:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821952, offset 192 00000100:00000200:10.0:1547251655.071701:0:13336:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821968, offset 192 00000100:00000200:2.0:1547251655.071702:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821936 00000400:00000200:11.0:1547251655.071703:0:12983:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:10.0:1547251655.071705:0:13336:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000400:00000200:11.0:1547251655.071707:0:12983:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000100:00000200:8.0:1547251655.071707:0:12982:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821984, offset 192 00010000:00000200:2.0:1547251655.071708:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692fed050 x1622383954821936/t0(0) o400->dc7951f7-3e71-e9c4-2e5e-5f95f4837d68@10.0.11.91@o2ib10:716/0 lens 224/224 e 0 to 0 dl 1547251661 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:11.0:1547251655.071710:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251655.071711:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251655.071711:0:12982:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:11.0:1547251655.071713:0:12983:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9764, 9766] 00000100:00000200:2.0:1547251655.071715:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954821936, offset 192 00000400:00000200:11.0:1547251655.071716:0:12983:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:2.0:1547251655.071717:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000400:00000200:2.0:1547251655.071721:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000400:00000200:11.0:1547251655.071723:0:12983:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000400:00000200:2.0:1547251655.071724:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251655.071726:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251655.071727:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9765, 9766] 00000800:00000200:11.0:1547251655.071728:0:12983:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:10.0:1547251655.071728:0:13336:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000400:00000200:2.0:1547251655.071729:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251655.071731:0:13336:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251655.071733:0:13336:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251655.071734:0:13336:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9765, 9767] 00000400:00000200:10.0:1547251655.071737:0:13336:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251655.071743:0:13336:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.1.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:10.0:1547251655.071747:0:13336:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251655.071748:0:12982:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251655.071750:0:12982:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251655.071752:0:12982:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251655.071753:0:12982:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9765, 9768] 00000400:00000200:8.0:1547251655.071755:0:12982:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:8.0:1547251655.071760:0:12982:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251655.071763:0:12982:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:11.0:1547251655.071765:0:12983:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251655.071767:0:12983:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:11.0:1547251655.071768:0:12983:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000400:00000200:2.0:1547251655.071768:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.1.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251655.071771:0:12983:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:2.0:1547251655.071771:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:10.0:1547251655.071772:0:13336:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251655.071772:0:12982:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:10.0:1547251655.071774:0:13336:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:8.0:1547251655.071774:0:12982:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:2.0:1547251655.071774:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:10.0:1547251655.071775:0:13336:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:8.0:1547251655.071775:0:12982:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (22)++ 00000800:00000200:2.0:1547251655.071776:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:8.0:1547251655.071777:0:12982:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251655.071778:0:13336:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:2.0:1547251655.071778:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:2.0:1547251655.071780:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:5.2:1547251655.071784:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (22)++ 00000100:00000200:2.0:1547251655.071784:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:11.0:1547251655.071800:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (23)++ 00000800:00000200:11.0:1547251655.071807:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (24)-- 00000400:00000200:11.0:1547251655.071809:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b908 00000400:00000200:11.0:1547251655.071811:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251655.071814:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251655.071814:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (23)++ 00000800:00000200:11.0:1547251655.071816:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (24)-- 00000800:00000200:10.0:1547251655.071816:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (24)-- 00000800:00000200:11.0:1547251655.071818:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:10.0:1547251655.071818:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688dd0 00000400:00000200:10.0:1547251655.071819:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251655.071822:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251655.071825:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:7.2:1547251655.071834:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:11.0:1547251655.071850:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:11.0:1547251655.071854:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (24)-- 00000400:00000200:11.0:1547251655.071856:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914dd0 00000400:00000200:11.0:1547251655.071858:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251655.071860:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)++ 00000400:00000200:11.0:1547251655.071861:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251655.071862:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (24)-- 00000800:00000200:11.0:1547251655.071863:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (24)-- 00000800:00000200:11.0:1547251655.071864:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:10.0:1547251655.071864:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a4c8 00000400:00000200:10.0:1547251655.071865:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251655.071868:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251655.071870:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:1.0:1547251655.146675:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446800, portal 4 00000100:00000200:1.0:1547251655.146678:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446800, offset 0 00000400:00000200:1.0:1547251655.146682:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:1.0:1547251655.146685:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:1.0:1547251655.146688:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23073] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251655.146691:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23072] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23073] 00000400:00000200:1.0:1547251655.146693:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:1.0:1547251655.146695:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:11.0:1547251655.146696:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446832, portal 4 00000400:00000200:1.0:1547251655.146696:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251655.146697:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9761, 9761] 00000100:00000200:11.0:1547251655.146699:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446832, offset 0 00000400:00000200:1.0:1547251655.146699:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:11.0:1547251655.146702:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:1.0:1547251655.146704:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000400:00000200:11.0:1547251655.146705:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000800:00000200:1.0:1547251655.146706:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:11.0:1547251655.146708:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23073] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:1.0:1547251655.146709:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000100:00000200:6.0:1547251655.146710:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446816, portal 4 00000400:00000200:11.0:1547251655.146711:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23073] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23073] 00000800:00000200:1.0:1547251655.146711:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:1.0:1547251655.146712:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:11.0:1547251655.146713:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000100:00000200:6.0:1547251655.146714:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446816, offset 0 00000400:00000200:11.0:1547251655.146715:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:1.0:1547251655.146715:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:11.0:1547251655.146716:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251655.146718:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9761, 9762] 00000400:00000200:6.0:1547251655.146718:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:11.0:1547251655.146719:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:6.0:1547251655.146721:0:12676:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:11.0:1547251655.146724:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:6.0:1547251655.146724:0:12676:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23074] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:11.0:1547251655.146727:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:6.0:1547251655.146727:0:12676:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23073] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23074] 00000400:00000200:6.0:1547251655.146729:0:12676:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:11.0:1547251655.146731:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000400:00000200:6.0:1547251655.146731:0:12676:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:11.0:1547251655.146732:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000400:00000200:6.0:1547251655.146732:0:12676:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251655.146733:0:12676:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9762, 9762] 00000800:00000200:11.0:1547251655.146734:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:6.0:1547251655.146735:0:12676:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000800:00000200:11.0:1547251655.146736:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:9.2:1547251655.146736:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:6.0:1547251655.146740:0:12676:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:6.0:1547251655.146743:0:12676:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:6.0:1547251655.146748:0:12676:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:6.0:1547251655.146749:0:12676:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:6.0:1547251655.146752:0:12676:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:7.2:1547251655.146755:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:6.0:1547251655.146755:0:12676:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (24)-- 00000800:00000200:15.0:1547251655.146790:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:15.0:1547251655.146793:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (24)-- 00000100:00000200:15.0:1547251655.146797:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b6c00 x1622384012446800/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251662 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251655.146802:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2110 00000400:00000200:15.0:1547251655.146803:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:15.0:1547251655.146806:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:15.0:1547251655.146808:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:15.0:1547251655.146810:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:15.0:1547251655.146811:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251655.146813:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000100:00000200:15.0:1547251655.146814:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07ad00 x1622384012446832/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251662 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251655.146816:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000400:00000200:15.0:1547251655.146818:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b7f8 00000400:00000200:15.0:1547251655.146819:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:12.0:1547251655.146820:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c436f00 x1622384012446816/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251662 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251655.146821:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:15.0:1547251655.146822:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:15.0:1547251655.146824:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:15.0:1547251655.146825:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:12.0:1547251655.146825:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990e58 00000400:00000200:12.0:1547251655.146826:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251655.146829:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251655.146832:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:7.2:1547251655.146895:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251655.146901:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:15.0:1547251655.146909:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:15.0:1547251655.146912:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000800:00000200:12.0:1547251655.146914:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:15.0:1547251655.146915:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251655.146917:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251655.146919:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04c50 00000400:00000200:12.0:1547251655.146920:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251655.146925:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fe9d [1] + 192 00000400:00000200:12.0:1547251655.146925:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04c70 00000400:00000200:12.0:1547251655.146928:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fead [1] + 192 00000100:00000200:15.0:1547251655.146933:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b6c00 x1622384012446800/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251662 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251655.146951:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:12.0:1547251655.146952:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07ad00 x1622384012446832/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251662 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251655.146953:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000800:00000200:15.0:1547251655.146954:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:15.0:1547251655.146955:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:15.0:1547251655.146957:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:15.0:1547251655.146958:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:15.0:1547251655.146961:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251655.146963:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:15.0:1547251655.146964:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04c60 00000800:00000200:12.0:1547251655.146965:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000100:00000200:1.0:1547251655.146966:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b6c00 x1622384012446800/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251662 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251655.146967:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:12.0:1547251655.146968:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251655.146970:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:12.0:1547251655.146971:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000100:00000200:1.0:1547251655.146974:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b6c00 x1622384012446800/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251662 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251655.146980:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00000400:00000200:15.0:1547251655.146982:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7febd [1] + 192 00010000:00080000:1.0:1547251655.146983:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b6c00 x1622384012446800/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251662 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251655.146986:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07ad00 x1622384012446832/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251662 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251655.146991:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07ad00 x1622384012446832/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251662 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251655.146995:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b908 00000100:00000200:15.0:1547251655.146997:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c436f00 x1622384012446816/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251662 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:11.0:1547251655.146999:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b17b07ad00 x1622384012446832/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251662 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:15.0:1547251655.147005:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251655.147006:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (21)++ 00000800:00000200:15.0:1547251655.147007:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (22)-- 00000800:00000200:15.0:1547251655.147008:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:6.0:1547251655.147015:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c436f00 x1622384012446816/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251662 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251655.147020:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c436f00 x1622384012446816/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251662 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251655.147025:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00010000:00080000:6.0:1547251655.147030:0:12676:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ae5c436f00 x1622384012446816/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251662 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251656.698695:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012446864, portal 10 00000100:00000200:4.0:1547251656.698699:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012446864, offset 0 00000400:00000200:4.0:1547251656.698702:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251656.698706:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251656.698709:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c04c90 00000400:00000200:4.0:1547251656.698713:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 63616 00000100:00000200:4.0:1547251656.698716:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:7.0:1547251656.698722:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012446896, portal 10 00000100:00000200:6.0:1547251656.698722:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012446880, portal 10 00000100:00000200:7.0:1547251656.698725:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012446896, offset 0 00000400:00000200:4.0:1547251656.698725:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251656.698726:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012446880, offset 0 00000400:00000200:7.0:1547251656.698729:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251656.698733:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251656.698733:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251656.698736:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:4.0:1547251656.698736:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c433600 x1622384012446864/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251656.698737:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c04cb0 00000400:00000200:6.0:1547251656.698739:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c04ca0 00000400:00000200:4.0:1547251656.698743:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bcc0 00000400:00000200:4.0:1547251656.698744:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251656.698745:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 63840 00000100:00000200:5.0:1547251656.698745:0:13338:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446864 00000400:00000200:4.0:1547251656.698746:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251656.698748:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 64064 00000100:00000200:7.0:1547251656.698751:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00010000:00000200:5.0:1547251656.698751:0:13338:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444ec00 x1622384012446864/t0(0) o400->cache1-MDT0000-lwp-MDT0000_UUID@0@lo:717/0 lens 224/224 e 0 to 0 dl 1547251662 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:4.0:1547251656.698754:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012446848, portal 25 00000400:00000200:7.0:1547251656.698757:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251656.698758:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:6.0:1547251656.698762:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251656.698763:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 26, xid 1622384012446848, offset 0 00000100:00000200:6.0:1547251656.698765:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c431b00 x1622384012446880/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251656.698771:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990908 00000400:00000200:6.0:1547251656.698772:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:5.0:1547251656.698773:0:13338:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012446864, offset 192 00000100:00000200:7.0:1547251656.698774:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c435a00 x1622384012446896/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251656.698774:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251656.698780:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea198 00000400:00000200:7.0:1547251656.698781:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251656.698783:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251656.698784:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251656.698786:0:13338:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251656.698786:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251656.698789:0:13338:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251656.698789:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 26 MB=0x5c38cb6c04c80 00000400:00000200:5.0:1547251656.698792:0:13338:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c04c90 00000400:00000200:4.0:1547251656.698792:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-0@lo of length 224/224 into md 0xd5529 [1] + 0 00000400:00000200:5.0:1547251656.698794:0:13338:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaf311 [1] + 192 00000400:00000200:4.0:1547251656.698795:0:12675:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d908 00000100:00000200:7.0:1547251656.698797:0:13355:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446880 00000100:00000200:6.0:1547251656.698797:0:12980:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446896 00000100:00000200:4.0:1547251656.698797:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000100:00000200:5.0:1547251656.698798:0:13338:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c433600 x1622384012446864/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00010000:00000200:7.0:1547251656.698802:0:13355:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a765df2700 x1622384012446880/t0(0) o400->cache1-MDT0000-lwp-OST0000_UUID@0@lo:717/0 lens 224/224 e 0 to 0 dl 1547251662 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:6.0:1547251656.698802:0:12980:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae37935100 x1622384012446896/t0(0) o400->cache1-MDT0000-lwp-OST0002_UUID@0@lo:717/0 lens 224/224 e 0 to 0 dl 1547251662 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:5.0:1547251656.698804:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251656.698805:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d908 00000400:00000200:5.0:1547251656.698806:0:13338:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad2a8 00000400:00000200:4.0:1547251656.698806:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251656.698807:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251656.698809:0:13338:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251656.698809:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c437200 x1622384012446848/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251663 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251656.698814:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b4c8 00000400:00000200:4.0:1547251656.698815:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:7.0:1547251656.698816:0:13355:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012446880, offset 192 00000100:00000200:6.0:1547251656.698816:0:12980:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012446896, offset 192 00000400:00000200:4.0:1547251656.698817:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251656.698818:0:13355:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251656.698820:0:12980:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:4.0:1547251656.698821:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012446912, portal 10 00000400:00000200:7.0:1547251656.698822:0:13355:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251656.698822:0:12980:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:4.0:1547251656.698823:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012446912, offset 0 00000400:00000200:7.0:1547251656.698825:0:13355:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c04ca0 00000400:00000200:6.0:1547251656.698825:0:12980:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c04cb0 00000400:00000200:4.0:1547251656.698826:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251656.698828:0:12980:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaf321 [1] + 192 00000400:00000200:4.0:1547251656.698829:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251656.698831:0:13355:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaf331 [1] + 192 00000400:00000200:4.0:1547251656.698832:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c04cc0 00000100:00000200:6.0:1547251656.698834:0:12980:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c435a00 x1622384012446896/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251656.698834:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 64288 00000100:00000200:4.0:1547251656.698837:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:6.0:1547251656.698840:0:12980:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251656.698841:0:13355:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c431b00 x1622384012446880/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251656.698847:0:12980:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990908 00000400:00000200:4.0:1547251656.698847:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251656.698848:0:13355:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251656.698848:0:12980:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:4.0:1547251656.698849:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c437800 x1622384012446912/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251656.698850:0:12980:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251656.698857:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b440 00000100:00000200:6.0:1547251656.698858:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c431b00 x1622384012446880/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251656.698858:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251656.698859:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251656.698860:0:13355:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea198 00000400:00000200:7.0:1547251656.698861:0:13355:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:5.0:1547251656.698862:0:13338:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446912 00000400:00000200:7.0:1547251656.698863:0:13355:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251656.698864:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c431b00 x1622384012446880/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251656.698864:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012446848 00010000:00000200:5.0:1547251656.698866:0:13338:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b824448600 x1622384012446912/t0(0) o400->cache1-MDT0000-lwp-OST0001_UUID@0@lo:717/0 lens 224/224 e 0 to 0 dl 1547251662 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:2.0:1547251656.698870:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692febc50 x1622384012446848/t0(0) o400->a65be517-33e7-fa36-0de3-da9e02df4f8d@0@lo:717/0 lens 224/224 e 0 to 0 dl 1547251662 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251656.698872:0:13338:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012446912, offset 192 00000400:00000200:5.0:1547251656.698874:0:13338:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251656.698876:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00000400:00000200:5.0:1547251656.698877:0:13338:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:4.0:1547251656.698878:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c433600 x1622384012446864/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251656.698880:0:13338:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c04cc0 00000100:00000200:4.0:1547251656.698884:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c433600 x1622384012446864/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251656.698889:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bc38 00000100:00000200:7.0:1547251656.698891:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c435a00 x1622384012446896/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251656.698896:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c435a00 x1622384012446896/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251656.698907:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea330 00000100:00000200:2.0:1547251656.698908:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384012446848, offset 192 00000400:00000200:5.0:1547251656.698910:0:13338:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaf351 [1] + 192 00000400:00000200:2.0:1547251656.698914:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:5.0:1547251656.698916:0:13338:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c437800 x1622384012446912/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251656.698917:0:12929:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:2.0:1547251656.698920:0:12929:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 25 MB=0x5c38cb6c04c80 00000400:00000200:5.0:1547251656.698925:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:2.0:1547251656.698925:0:12929:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 19 from 12345-0@lo of length 192/192 into md 0xaf341 [1] + 192 00000400:00000200:5.0:1547251656.698927:0:13338:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad2a8 00000400:00000200:5.0:1547251656.698928:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:2.0:1547251656.698929:0:12929:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c437200 x1622384012446848/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251656.698930:0:13338:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:2.0:1547251656.698934:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251656.698935:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c437200 x1622384012446848/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251656.698941:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c437200 x1622384012446848/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251656.698956:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bcc0 00000400:00000200:2.0:1547251656.698958:0:12929:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06ab28 00000400:00000200:2.0:1547251656.698958:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:4.0:1547251656.698960:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c437800 x1622384012446912/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251656.698960:0:12929:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:2.0:1547251656.698964:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000100:00000200:4.0:1547251656.698966:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c437800 x1622384012446912/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251663 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251656.698972:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b4c8 00000100:00000200:1.0:1547251657.258682:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446928, portal 4 00000100:00000200:15.0:1547251657.258683:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446944, portal 4 00000100:00000200:1.0:1547251657.258685:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446928, offset 0 00000100:00000200:15.0:1547251657.258686:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446944, offset 0 00000400:00000200:1.0:1547251657.258688:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:15.0:1547251657.258689:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:1.0:1547251657.258691:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:15.0:1547251657.258692:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:1.0:1547251657.258693:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23074] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251657.258694:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23074] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251657.258696:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23074] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23074] 00000400:00000200:15.0:1547251657.258697:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23074] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23074] 00000400:00000200:1.0:1547251657.258699:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:15.0:1547251657.258700:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:15.0:1547251657.258701:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:9.0:1547251657.258701:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446960, portal 4 00000400:00000200:1.0:1547251657.258701:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251657.258703:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:9.0:1547251657.258703:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446960, offset 0 00000400:00000200:1.0:1547251657.258703:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251657.258704:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9766, 9768] 00000400:00000200:1.0:1547251657.258705:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9766, 9768] 00000400:00000200:15.0:1547251657.258706:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:9.0:1547251657.258707:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:1.0:1547251657.258707:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:15.0:1547251657.258711:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251657.258714:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:1.0:1547251657.258717:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251657.258718:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251657.258719:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:15.0:1547251657.258720:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000400:00000200:9.0:1547251657.258720:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:1.0:1547251657.258720:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:15.0:1547251657.258723:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)-- 00000400:00000200:9.0:1547251657.258723:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:254, d:10, s:23076] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:1.0:1547251657.258723:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251657.258724:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:1.0:1547251657.258725:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (22)++ 00000400:00000200:9.0:1547251657.258726:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23074] with best_ni 10.0.11.90@o2ib10 [c:254, d:10, s:23076] 00000800:00000200:1.0:1547251657.258728:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (23)-- 00000400:00000200:9.0:1547251657.258729:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251657.258731:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251657.258732:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251657.258733:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[6, 8], s:[9768, 9768] 00000400:00000200:9.0:1547251657.258735:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:9.0:1547251657.258739:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251657.258742:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:9.0:1547251657.258745:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251657.258746:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:9.0:1547251657.258747:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:9.0:1547251657.258750:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:5.2:1547251657.258772:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251657.258788:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (23)++ 00000800:00000200:11.0:1547251657.258797:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (24)-- 00000800:00000200:7.2:1547251657.258799:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000100:00000200:11.0:1547251657.258802:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8ca9800 x1622384012446944/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251657.258810:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1cc0 00000400:00000200:11.0:1547251657.258811:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251657.258815:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251657.258815:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (23)++ 00000800:00000200:10.0:1547251657.258817:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (24)-- 00000800:00000200:11.0:1547251657.258818:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (24)-- 00000800:00000200:11.0:1547251657.258820:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000100:00000200:10.0:1547251657.258820:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b6900 x1622384012446928/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251657.258821:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000100:00000200:11.0:1547251657.258824:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf3f00 x1622384012446960/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251657.258827:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e22a8 00000400:00000200:10.0:1547251657.258828:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251657.258830:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea5d8 00000400:00000200:11.0:1547251657.258831:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251657.258831:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251657.258833:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:11.0:1547251657.258835:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251657.258835:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:11.0:1547251657.258836:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251657.258838:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:5.2:1547251657.258866:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:11.0:1547251657.258878:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251657.258885:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251657.258887:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251657.258889:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251657.258890:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (23)++ 00000800:00000200:11.0:1547251657.258891:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251657.258892:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251657.258892:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000800:00000200:7.2:1547251657.258895:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000400:00000200:10.0:1547251657.258896:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251657.258901:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04ce0 00000800:00000200:11.0:1547251657.258904:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:10.0:1547251657.258905:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7389 [1] + 192 00000800:00000200:5.2:1547251657.258905:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251657.258907:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251657.258911:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000100:00000200:10.0:1547251657.258911:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8ca9800 x1622384012446944/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251657.258915:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04cf0 00000400:00000200:10.0:1547251657.258922:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:11.0:1547251657.258924:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7399 [1] + 192 00000800:00000200:10.0:1547251657.258924:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:10.0:1547251657.258926:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251657.258927:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000100:00000200:11.0:1547251657.258928:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf3f00 x1622384012446960/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251657.258929:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251657.258931:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251657.258933:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251657.258937:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251657.258938:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251657.258940:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (21)++ 00000100:00000200:15.0:1547251657.258941:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8ca9800 x1622384012446944/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251657.258941:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04cd0 00000800:00000200:11.0:1547251657.258950:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251657.258951:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:15.0:1547251657.258953:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8ca9800 x1622384012446944/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251657.258953:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:15.0:1547251657.258958:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb15d8 00010000:00080000:15.0:1547251657.258961:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8ca9800 x1622384012446944/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251657.258961:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7379 [1] + 192 00000100:00000200:9.0:1547251657.258966:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf3f00 x1622384012446960/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251657.258972:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf3f00 x1622384012446960/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251657.258976:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea000 00000100:00000200:10.0:1547251657.258978:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b6900 x1622384012446928/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251657.258979:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf3f00 x1622384012446960/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251657.258988:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251657.258989:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251657.258990:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251657.258991:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:1.0:1547251657.259006:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b6900 x1622384012446928/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:1.0:1547251657.259011:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b6900 x1622384012446928/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251657.259016:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00010000:00080000:1.0:1547251657.259020:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b6900 x1622384012446928/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:15.0:1547251657.722676:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446976, portal 4 00000100:00000200:15.0:1547251657.722679:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446976, offset 0 00000400:00000200:15.0:1547251657.722682:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:15.0:1547251657.722685:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:15.0:1547251657.722687:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23076] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251657.722691:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23075] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23076] 00000400:00000200:15.0:1547251657.722693:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:15.0:1547251657.722694:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251657.722696:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251657.722697:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9760, 9761] 00000400:00000200:15.0:1547251657.722699:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000100:00000200:0.0:1547251657.722702:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447008, portal 4 00000400:00000200:15.0:1547251657.722703:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000100:00000200:0.0:1547251657.722705:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447008, offset 0 00000800:00000200:15.0:1547251657.722706:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000100:00000200:9.0:1547251657.722707:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012446992, portal 4 00000800:00000200:15.0:1547251657.722709:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:0.0:1547251657.722709:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:15.0:1547251657.722710:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000100:00000200:9.0:1547251657.722710:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012446992, offset 0 00000800:00000200:15.0:1547251657.722711:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:0.0:1547251657.722712:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251657.722713:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:15.0:1547251657.722714:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:0.0:1547251657.722714:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23076] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251657.722716:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251657.722718:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23076] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251657.722718:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23076] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23076] 00000400:00000200:9.0:1547251657.722721:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23076] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23076] 00000800:00000200:0.2:1547251657.722722:0:12670:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:9.0:1547251657.722723:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251657.722725:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251657.722726:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251657.722727:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9761, 9761] 00000400:00000200:9.0:1547251657.722729:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:0.0:1547251657.722731:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:0.0:1547251657.722733:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251657.722734:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:0.0:1547251657.722735:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251657.722736:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9761, 9762] 00000400:00000200:0.0:1547251657.722738:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:9.0:1547251657.722739:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:9.0:1547251657.722742:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251657.722743:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000400:00000200:0.0:1547251657.722743:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251657.722744:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:0.0:1547251657.722746:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:9.0:1547251657.722747:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:0.0:1547251657.722749:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251657.722750:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:0.0:1547251657.722751:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:0.0:1547251657.722754:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:12.2:1547251657.722762:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:12.2:1547251657.722771:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251657.722776:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251657.722781:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:4.0:1547251657.722786:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cae300 x1622384012446976/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251657.722789:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251657.722791:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000400:00000200:4.0:1547251657.722794:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1088 00000100:00000200:5.0:1547251657.722795:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf5700 x1622384012446992/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251657.722795:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251657.722799:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:5.0:1547251657.722802:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaaa0 00000800:00000200:4.0:1547251657.722802:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:5.0:1547251657.722803:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251657.722804:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251657.722805:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000400:00000200:5.0:1547251657.722806:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000100:00000200:4.0:1547251657.722808:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328de600 x1622384012447008/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251657.722809:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251657.722811:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:5.0:1547251657.722812:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:5.0:1547251657.722814:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:4.0:1547251657.722814:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665440 00000400:00000200:4.0:1547251657.722815:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251657.722818:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251657.722821:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:12.2:1547251657.722893:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:0.2:1547251657.722897:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:5.0:1547251657.722906:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251657.722909:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251657.722910:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:12.2:1547251657.722912:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:4.0:1547251657.722913:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251657.722915:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251657.722917:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251657.722920:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04d20 00000400:00000200:4.0:1547251657.722921:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04d00 00000400:00000200:5.0:1547251657.722924:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe42c5 [1] + 192 00000400:00000200:4.0:1547251657.722929:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe42b5 [1] + 192 00000100:00000200:5.0:1547251657.722933:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328de600 x1622384012447008/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251657.722954:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251657.722956:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000100:00000200:4.0:1547251657.722956:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cae300 x1622384012446976/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251657.722958:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251657.722960:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251657.722962:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:5.0:1547251657.722963:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251657.722965:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251657.722967:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251657.722967:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251657.722969:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (21)++ 00000100:00000200:0.0:1547251657.722969:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328de600 x1622384012447008/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251657.722971:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:5.0:1547251657.722972:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251657.722972:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:4.0:1547251657.722974:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:0.0:1547251657.722975:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328de600 x1622384012447008/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251657.722977:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04d10 00000400:00000200:0.0:1547251657.722979:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00000100:00000200:15.0:1547251657.722982:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cae300 x1622384012446976/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251657.722982:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328de600 x1622384012447008/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:15.0:1547251657.722988:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cae300 x1622384012446976/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251657.722994:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb15d8 00010000:00080000:15.0:1547251657.722997:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cae300 x1622384012446976/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251657.722997:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe42d5 [1] + 192 00000100:00000200:5.0:1547251657.723000:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf5700 x1622384012446992/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251657.723008:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251657.723009:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251657.723010:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251657.723011:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:9.0:1547251657.723025:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf5700 x1622384012446992/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251657.723031:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf5700 x1622384012446992/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251657.723035:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea000 00010000:00080000:9.0:1547251657.723039:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf5700 x1622384012446992/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251664 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251658.122696:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447024, portal 4 00000100:00000200:9.0:1547251658.122698:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447024, offset 0 00000400:00000200:9.0:1547251658.122701:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251658.122704:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251658.122706:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04d30 00000400:00000200:9.0:1547251658.122709:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4b69 [1] + 0 00000400:00000200:9.0:1547251658.122711:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d688660 00000100:00000200:9.0:1547251658.122713:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251658.122722:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688660 00000400:00000200:9.0:1547251658.122723:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251658.122727:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf2700 x1622384012447024/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251665 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251658.122733:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea3b8 00000400:00000200:9.0:1547251658.122734:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251658.122735:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251658.122744:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447024 00000100:00000200:6.0:1547251658.122748:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447040, portal 4 00000100:00000200:6.0:1547251658.122752:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447040, offset 0 00010000:00000200:8.0:1547251658.122754:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3d9050 x1622384012447024/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:719/0 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:6.0:1547251658.122755:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251658.122759:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:8.0:1547251658.122762:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447024, offset 192 00000400:00000200:6.0:1547251658.122762:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04d40 00000400:00000200:8.0:1547251658.122764:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251658.122765:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2ab5 [1] + 0 00000400:00000200:8.0:1547251658.122767:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251658.122767:0:12675:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040b3b8 00000400:00000200:8.0:1547251658.122769:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04d30 00000100:00000200:6.0:1547251658.122769:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251658.122771:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf391 [1] + 192 00000100:00000200:8.0:1547251658.122774:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf2700 x1622384012447024/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251665 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251658.122778:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b3b8 00000400:00000200:6.0:1547251658.122779:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251658.122782:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251658.122782:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b82444e000 x1622384012447040/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251665 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251658.122785:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688110 00000400:00000200:8.0:1547251658.122786:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251658.122787:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990908 00000400:00000200:8.0:1547251658.122788:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251658.122788:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251658.122789:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251658.122792:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf2700 x1622384012447024/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251665 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251658.122795:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:9.0:1547251658.122798:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf2700 x1622384012447024/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251665 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251658.122799:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447040 00000400:00000200:9.0:1547251658.122803:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea000 00010000:00080000:9.0:1547251658.122808:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf2700 x1622384012447024/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251665 ref 1 fl Rpc:R/0/0 rc 0/0 00010000:00000200:4.0:1547251658.122811:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11fc50 x1622384012447040/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:719/0 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251658.122821:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447040, offset 192 00000400:00000200:4.0:1547251658.122824:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251658.122827:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251658.122830:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04d40 00000400:00000200:4.0:1547251658.122833:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf3a1 [1] + 192 00000100:00000200:4.0:1547251658.122836:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b82444e000 x1622384012447040/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251665 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251658.122845:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251658.122847:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b4c8 00000400:00000200:4.0:1547251658.122848:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251658.122849:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251658.122854:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b82444e000 x1622384012447040/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251665 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251658.122855:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251658.122858:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b82444e000 x1622384012447040/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251665 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251658.122862:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00010000:00080000:6.0:1547251658.122865:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b82444e000 x1622384012447040/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251665 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251658.138734:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447056, portal 4 00000100:00000200:7.0:1547251658.138738:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447056, offset 0 00000400:00000200:7.0:1547251658.138741:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251658.138744:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251658.138747:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04d50 00000400:00000200:7.0:1547251658.138749:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2ae5 [1] + 0 00000400:00000200:7.0:1547251658.138751:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad990 00000100:00000200:7.0:1547251658.138753:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:7.0:1547251658.138760:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad990 00000400:00000200:7.0:1547251658.138761:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251658.138763:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c430300 x1622384012447056/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251665 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251658.138768:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea198 00000400:00000200:7.0:1547251658.138768:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251658.138769:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251658.138790:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447056 00010000:00000200:4.0:1547251658.138801:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11f850 x1622384012447056/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:719/0 lens 224/368 e 0 to 0 dl 1547251664 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251658.138811:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447056, offset 192 00000400:00000200:4.0:1547251658.138815:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251658.138818:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251658.138821:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04d50 00000400:00000200:4.0:1547251658.138824:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf3b1 [1] + 192 00000100:00000200:4.0:1547251658.138827:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c430300 x1622384012447056/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251665 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251658.138837:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251658.138839:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bcc0 00000400:00000200:4.0:1547251658.138840:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251658.138841:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251658.138845:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c430300 x1622384012447056/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251665 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251658.138846:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251658.138849:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c430300 x1622384012447056/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251665 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251658.138854:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea330 00010000:00080000:7.0:1547251658.138857:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae5c430300 x1622384012447056/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251665 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251660.154681:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447072, portal 4 00000100:00000200:7.0:1547251660.154684:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447072, offset 0 00000400:00000200:7.0:1547251660.154687:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:7.0:1547251660.154690:0:12677:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:7.0:1547251660.154692:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23078] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:7.0:1547251660.154695:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23076] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23078] 00000400:00000200:7.0:1547251660.154697:0:12677:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:7.0:1547251660.154699:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251660.154701:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251660.154702:0:12677:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9762, 9763] 00000400:00000200:7.0:1547251660.154704:0:12677:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000100:00000200:11.0:1547251660.154705:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447088, portal 4 00000100:00000200:11.0:1547251660.154708:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447088, offset 0 00000400:00000200:7.0:1547251660.154709:0:12677:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:11.0:1547251660.154711:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:7.0:1547251660.154712:0:12677:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:11.0:1547251660.154714:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000100:00000200:0.0:1547251660.154714:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447104, portal 4 00000800:00000200:7.0:1547251660.154715:0:12677:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000400:00000200:11.0:1547251660.154716:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23078] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:7.0:1547251660.154716:0:12677:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:7.0:1547251660.154717:0:12677:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000100:00000200:0.0:1547251660.154717:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447104, offset 0 00000400:00000200:11.0:1547251660.154720:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23077] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23078] 00000800:00000200:7.0:1547251660.154720:0:12677:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:0.0:1547251660.154720:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:11.0:1547251660.154722:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:11.0:1547251660.154723:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251660.154723:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:11.0:1547251660.154725:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251660.154725:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23079] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:11.0:1547251660.154726:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9763, 9763] 00000400:00000200:11.0:1547251660.154728:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:0.0:1547251660.154729:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23077] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23079] 00000400:00000200:0.0:1547251660.154731:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:0.0:1547251660.154732:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251660.154733:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:9.2:1547251660.154734:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:0.0:1547251660.154735:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:11.0:1547251660.154736:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:0.0:1547251660.154737:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9763, 9764] 00000800:00000200:11.0:1547251660.154739:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000400:00000200:0.0:1547251660.154739:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000800:00000200:11.0:1547251660.154740:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:11.0:1547251660.154742:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000400:00000200:0.0:1547251660.154743:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251660.154744:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:0.0:1547251660.154746:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:0.0:1547251660.154749:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251660.154751:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:0.0:1547251660.154752:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:0.0:1547251660.154755:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:7.2:1547251660.154761:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000800:00000200:7.2:1547251660.154768:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251660.154788:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:12.0:1547251660.154791:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000100:00000200:12.0:1547251660.154795:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df0c00 x1622384012447072/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251667 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251660.154800:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea198 00000400:00000200:12.0:1547251660.154801:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251660.154804:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251660.154806:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:12.0:1547251660.154808:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:12.0:1547251660.154811:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000100:00000200:12.0:1547251660.154813:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07d400 x1622384012447088/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251667 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251660.154815:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000400:00000200:12.0:1547251660.154817:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b000 00000800:00000200:15.0:1547251660.154818:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000400:00000200:12.0:1547251660.154818:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251660.154820:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000100:00000200:15.0:1547251660.154821:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328de900 x1622384012447104/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251667 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251660.154821:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251660.154823:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:12.0:1547251660.154824:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:12.0:1547251660.154826:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:15.0:1547251660.154827:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00000400:00000200:15.0:1547251660.154828:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:15.0:1547251660.154831:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:15.0:1547251660.154833:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:7.2:1547251660.154852:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:12.0:1547251660.154864:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251660.154874:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.1.11.93@o2ib10 00000800:00000200:12.0:1547251660.154875:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251660.154876:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251660.154877:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251660.154878:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:7.2:1547251660.154889:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:9.2:1547251660.154896:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:9.2:1547251660.154902:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:12.0:1547251660.154903:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251660.154905:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000800:00000200:15.0:1547251660.154908:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:12.0:1547251660.154909:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:15.0:1547251660.154911:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251660.154912:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04d60 00000400:00000200:15.0:1547251660.154915:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251660.154916:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7feed [1] + 192 00000400:00000200:15.0:1547251660.154919:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04d80 00000100:00000200:12.0:1547251660.154919:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df0c00 x1622384012447072/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251667 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251660.154927:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:15.0:1547251660.154928:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7ff0d [1] + 192 00000800:00000200:12.0:1547251660.154929:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:12.0:1547251660.154930:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:12.0:1547251660.154931:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:12.0:1547251660.154932:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000100:00000200:7.0:1547251660.154933:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df0c00 x1622384012447072/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251667 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251660.154934:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.93@o2ib10 00000400:00000200:12.0:1547251660.154937:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000100:00000200:7.0:1547251660.154938:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df0c00 x1622384012447072/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251667 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251660.154939:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04d70 00000400:00000200:7.0:1547251660.154960:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea330 00000100:00000200:15.0:1547251660.154962:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328de900 x1622384012447104/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251667 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:7.0:1547251660.154964:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a765df0c00 x1622384012447072/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251667 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:15.0:1547251660.154974:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:12.0:1547251660.154975:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fefd [1] + 192 00000800:00000200:15.0:1547251660.154976:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:15.0:1547251660.154977:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:15.0:1547251660.154978:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:12.0:1547251660.154978:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07d400 x1622384012447088/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251667 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251660.154980:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:15.0:1547251660.154981:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:15.0:1547251660.154983:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:12.0:1547251660.154986:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251660.154988:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251660.154989:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:0.0:1547251660.154989:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328de900 x1622384012447104/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251667 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251660.154990:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:0.0:1547251660.154995:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328de900 x1622384012447104/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251667 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251660.154999:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00000100:00000200:11.0:1547251660.155002:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07d400 x1622384012447088/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251667 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251660.155003:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328de900 x1622384012447104/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251667 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251660.155012:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07d400 x1622384012447088/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251667 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251660.155016:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b908 00010000:00080000:11.0:1547251660.155020:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b17b07d400 x1622384012447088/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251667 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251662.266687:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447120, portal 4 00000100:00000200:8.0:1547251662.266690:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447120, offset 0 00000400:00000200:8.0:1547251662.266693:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251662.266695:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251662.266698:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23080] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251662.266701:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23077] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23080] 00000400:00000200:8.0:1547251662.266703:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:8.0:1547251662.266705:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251662.266706:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251662.266707:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9768, 9769] 00000400:00000200:8.0:1547251662.266709:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000100:00000200:14.0:1547251662.266712:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447136, portal 4 00000400:00000200:8.0:1547251662.266713:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000100:00000200:14.0:1547251662.266715:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447136, offset 0 00000800:00000200:8.0:1547251662.266716:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:14.0:1547251662.266718:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:8.0:1547251662.266719:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251662.266720:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000400:00000200:14.0:1547251662.266721:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:8.0:1547251662.266721:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000400:00000200:14.0:1547251662.266723:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23080] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:8.0:1547251662.266724:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000400:00000200:14.0:1547251662.266726:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23078] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23080] 00000400:00000200:14.0:1547251662.266729:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:14.0:1547251662.266730:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251662.266732:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251662.266733:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9769, 9769] 00000400:00000200:14.0:1547251662.266735:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:14.0:1547251662.266739:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251662.266742:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:14.0:1547251662.266745:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251662.266747:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:14.0:1547251662.266748:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:14.0:1547251662.266751:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000100:00000200:2.0:1547251662.266760:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447152, portal 4 00000100:00000200:2.0:1547251662.266764:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447152, offset 0 00000400:00000200:2.0:1547251662.266768:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:2.0:1547251662.266771:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:7.2:1547251662.266774:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000400:00000200:2.0:1547251662.266774:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23081] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251662.266777:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23078] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23081] 00000400:00000200:2.0:1547251662.266779:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251662.266782:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251662.266783:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251662.266784:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9769, 9770] 00000400:00000200:2.0:1547251662.266786:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251662.266790:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000400:00000200:2.0:1547251662.266791:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251662.266794:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:2.0:1547251662.266797:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251662.266798:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (23)++ 00000800:00000200:11.0:1547251662.266800:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000800:00000200:2.0:1547251662.266800:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (24)++ 00000800:00000200:2.0:1547251662.266802:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (24)-- 00000800:00000200:5.2:1547251662.266803:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000100:00000200:11.0:1547251662.266804:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824753f00 x1622384012447120/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251662.266812:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688f68 00000400:00000200:11.0:1547251662.266814:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251662.266817:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251662.266817:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:10.0:1547251662.266819:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000800:00000200:11.0:1547251662.266820:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (24)-- 00000800:00000200:11.0:1547251662.266822:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000100:00000200:10.0:1547251662.266822:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328dec00 x1622384012447152/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251662.266824:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000100:00000200:11.0:1547251662.266827:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a0c00 x1622384012447136/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251662.266829:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06aa18 00000400:00000200:10.0:1547251662.266830:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251662.266833:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:10.0:1547251662.266833:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251662.266834:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251662.266835:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:11.0:1547251662.266837:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251662.266837:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:10.0:1547251662.266838:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251662.266839:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:7.2:1547251662.266890:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:10.0:1547251662.266903:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251662.266911:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251662.266915:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:10.0:1547251662.266916:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:7.2:1547251662.266919:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000400:00000200:10.0:1547251662.266921:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04d90 00000800:00000200:5.2:1547251662.266924:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000400:00000200:10.0:1547251662.266925:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7409 [1] + 192 00000800:00000200:11.0:1547251662.266926:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251662.266929:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000100:00000200:10.0:1547251662.266930:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824753f00 x1622384012447120/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251662.266933:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251662.266937:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04da0 00000400:00000200:10.0:1547251662.266941:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:11.0:1547251662.266952:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7429 [1] + 192 00000800:00000200:10.0:1547251662.266952:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251662.266954:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251662.266955:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:11.0:1547251662.266957:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a0c00 x1622384012447136/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251662.266957:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251662.266959:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251662.266963:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251662.266967:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:10.0:1547251662.266967:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04db0 00000800:00000200:11.0:1547251662.266969:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000100:00000200:8.0:1547251662.266970:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824753f00 x1622384012447120/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251662.266972:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:11.0:1547251662.266973:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251662.266975:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:11.0:1547251662.266976:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000100:00000200:8.0:1547251662.266976:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824753f00 x1622384012447120/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251662.266980:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688908 00000400:00000200:10.0:1547251662.266984:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7439 [1] + 192 00010000:00080000:8.0:1547251662.266984:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824753f00 x1622384012447120/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:14.0:1547251662.266989:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a0c00 x1622384012447136/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:14.0:1547251662.266995:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a0c00 x1622384012447136/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251662.266999:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00010000:00080000:14.0:1547251662.267002:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a0c00 x1622384012447136/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251662.267002:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328dec00 x1622384012447152/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251662.267011:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251662.267012:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251662.267013:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251662.267014:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:2.0:1547251662.267021:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328dec00 x1622384012447152/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251662.267026:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328dec00 x1622384012447152/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251662.267030:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a5d8 00010000:00080000:2.0:1547251662.267034:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328dec00 x1622384012447152/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251662.730687:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447168, portal 4 00000100:00000200:8.0:1547251662.730689:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447168, offset 0 00000400:00000200:8.0:1547251662.730692:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251662.730695:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251662.730697:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23081] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251662.730701:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23079] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23081] 00000400:00000200:8.0:1547251662.730703:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:8.0:1547251662.730704:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251662.730706:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251662.730707:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9762, 9762] 00000400:00000200:8.0:1547251662.730709:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:8.0:1547251662.730713:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251662.730716:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:8.0:1547251662.730719:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251662.730720:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:8.0:1547251662.730721:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000100:00000200:1.0:1547251662.730721:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447200, portal 4 00000800:00000200:8.0:1547251662.730724:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000100:00000200:1.0:1547251662.730724:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447200, offset 0 00000400:00000200:1.0:1547251662.730728:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:1.0:1547251662.730731:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000100:00000200:15.0:1547251662.730734:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447184, portal 4 00000400:00000200:1.0:1547251662.730735:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23081] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:15.0:1547251662.730738:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447184, offset 0 00000400:00000200:1.0:1547251662.730740:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23080] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23081] 00000400:00000200:15.0:1547251662.730741:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:0.2:1547251662.730741:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000400:00000200:1.0:1547251662.730743:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:15.0:1547251662.730744:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:1.0:1547251662.730745:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251662.730746:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23082] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251662.730747:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251662.730749:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9762, 9763] 00000400:00000200:15.0:1547251662.730750:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23080] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23082] 00000400:00000200:1.0:1547251662.730752:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:15.0:1547251662.730753:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:15.0:1547251662.730755:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251662.730757:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251662.730758:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9763, 9763] 00000400:00000200:15.0:1547251662.730760:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:1.0:1547251662.730760:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:1.0:1547251662.730764:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:15.0:1547251662.730766:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:1.0:1547251662.730768:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251662.730769:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:1.0:1547251662.730771:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:15.0:1547251662.730772:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251662.730772:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:15.0:1547251662.730774:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:15.0:1547251662.730775:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:1.0:1547251662.730775:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:15.0:1547251662.730778:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:12.2:1547251662.730791:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000800:00000200:12.2:1547251662.730799:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251662.730801:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251662.730805:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000100:00000200:4.0:1547251662.730811:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824750900 x1622384012447168/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251662.730816:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000400:00000200:4.0:1547251662.730818:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688aa0 00000800:00000200:5.0:1547251662.730819:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000400:00000200:4.0:1547251662.730820:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:5.0:1547251662.730822:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328def00 x1622384012447200/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251662.730823:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251662.730826:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251662.730828:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000400:00000200:5.0:1547251662.730829:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2a18 00000400:00000200:5.0:1547251662.730830:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251662.730830:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000400:00000200:5.0:1547251662.730833:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000100:00000200:4.0:1547251662.730833:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a0f00 x1622384012447184/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251662.730836:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251662.730838:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000400:00000200:4.0:1547251662.730838:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1088 00000800:00000200:5.0:1547251662.730839:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000400:00000200:4.0:1547251662.730840:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:5.0:1547251662.730841:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:4.0:1547251662.730843:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251662.730845:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:12.2:1547251662.730903:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:0.2:1547251662.730908:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:0.2:1547251662.730913:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:5.0:1547251662.730916:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251662.730920:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000800:00000200:4.0:1547251662.730921:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251662.730923:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251662.730925:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251662.730928:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251662.730931:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04dd0 00000400:00000200:4.0:1547251662.730931:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04dc0 00000400:00000200:5.0:1547251662.730935:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4375 [1] + 192 00000400:00000200:4.0:1547251662.730939:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4355 [1] + 192 00000100:00000200:5.0:1547251662.730953:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a0f00 x1622384012447184/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251662.730965:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251662.730967:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824750900 x1622384012447168/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251662.730968:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251662.730970:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251662.730971:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251662.730973:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251662.730975:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251662.730980:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251662.730981:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000100:00000200:15.0:1547251662.730982:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a0f00 x1622384012447184/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251662.730982:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251662.730983:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000400:00000200:5.0:1547251662.730984:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04de0 00000800:00000200:4.0:1547251662.730985:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000100:00000200:15.0:1547251662.730987:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a0f00 x1622384012447184/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251662.730987:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:4.0:1547251662.730988:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251662.730990:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:15.0:1547251662.730992:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb15d8 00010000:00080000:15.0:1547251662.730995:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a0f00 x1622384012447184/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251662.730995:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4365 [1] + 192 00000100:00000200:8.0:1547251662.731000:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824750900 x1622384012447168/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251662.731006:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824750900 x1622384012447168/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251662.731010:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688908 00010000:00080000:8.0:1547251662.731013:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824750900 x1622384012447168/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251662.731013:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328def00 x1622384012447200/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251662.731020:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251662.731022:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251662.731023:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251662.731024:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:1.0:1547251662.731031:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328def00 x1622384012447200/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:1.0:1547251662.731037:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328def00 x1622384012447200/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251662.731041:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00010000:00080000:1.0:1547251662.731045:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328def00 x1622384012447200/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251669 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251663.130682:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447216, portal 4 00000100:00000200:8.0:1547251663.130686:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447216, offset 0 00000400:00000200:8.0:1547251663.130688:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251663.130691:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251663.130693:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04df0 00000400:00000200:8.0:1547251663.130696:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4bd9 [1] + 0 00000400:00000200:8.0:1547251663.130698:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ae6a9145d8 00000100:00000200:8.0:1547251663.130700:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251663.130708:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a9145d8 00000400:00000200:8.0:1547251663.130710:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251663.130711:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447232, portal 4 00000100:00000200:8.0:1547251663.130713:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824752d00 x1622384012447216/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251670 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251663.130714:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447232, offset 0 00000400:00000200:7.0:1547251663.130716:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251663.130719:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251663.130720:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6882a8 00000400:00000200:8.0:1547251663.130721:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251663.130722:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251663.130722:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04e00 00000400:00000200:7.0:1547251663.130725:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2b55 [1] + 0 00000400:00000200:7.0:1547251663.130727:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad440 00000100:00000200:7.0:1547251663.130729:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:9.0:1547251663.130730:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447216 00000400:00000200:7.0:1547251663.130737:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad440 00000400:00000200:7.0:1547251663.130738:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00010000:00000200:9.0:1547251663.130740:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3d8c50 x1622384012447216/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:724/0 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:7.0:1547251663.130741:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df6600 x1622384012447232/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251670 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251663.130746:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea770 00000400:00000200:7.0:1547251663.130746:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:9.0:1547251663.130747:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447216, offset 192 00000400:00000200:7.0:1547251663.130747:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:9.0:1547251663.130749:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251663.130752:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251663.130754:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04df0 00000400:00000200:9.0:1547251663.130757:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf3f1 [1] + 192 00000100:00000200:9.0:1547251663.130759:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824752d00 x1622384012447216/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251670 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251663.130767:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251663.130769:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea000 00000100:00000200:5.0:1547251663.130769:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447232 00000400:00000200:9.0:1547251663.130770:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251663.130773:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251663.130777:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824752d00 x1622384012447216/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251670 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00000200:5.0:1547251663.130778:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11f450 x1622384012447232/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:724/0 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:9.0:1547251663.130779:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:8.0:1547251663.130783:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824752d00 x1622384012447216/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251670 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251663.130785:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447232, offset 192 00000400:00000200:5.0:1547251663.130787:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251663.130789:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688908 00000400:00000200:5.0:1547251663.130790:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00010000:00080000:8.0:1547251663.130792:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824752d00 x1622384012447216/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251670 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251663.130793:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04e00 00000400:00000200:5.0:1547251663.130796:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf401 [1] + 192 00000100:00000200:5.0:1547251663.130799:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df6600 x1622384012447232/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251670 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251663.130808:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251663.130810:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad2a8 00000400:00000200:5.0:1547251663.130811:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251663.130812:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251663.130817:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df6600 x1622384012447232/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251670 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251663.130817:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251663.130822:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df6600 x1622384012447232/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251670 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251663.130827:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea330 00010000:00080000:7.0:1547251663.130830:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df6600 x1622384012447232/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251670 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251663.146690:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447248, portal 4 00000100:00000200:6.0:1547251663.146693:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447248, offset 0 00000400:00000200:6.0:1547251663.146696:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251663.146699:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251663.146701:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04e10 00000400:00000200:6.0:1547251663.146704:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2b85 [1] + 0 00000400:00000200:6.0:1547251663.146706:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad770 00000100:00000200:6.0:1547251663.146708:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251663.146715:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad770 00000400:00000200:6.0:1547251663.146718:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251663.146720:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37934b00 x1622384012447248/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251670 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251663.146725:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990908 00000400:00000200:6.0:1547251663.146725:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251663.146727:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251663.146746:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447248 00010000:00000200:5.0:1547251663.146755:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11f050 x1622384012447248/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:724/0 lens 224/368 e 0 to 0 dl 1547251669 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251663.146762:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447248, offset 192 00000400:00000200:5.0:1547251663.146764:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251663.146767:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251663.146770:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04e10 00000400:00000200:5.0:1547251663.146773:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf411 [1] + 192 00000100:00000200:5.0:1547251663.146777:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37934b00 x1622384012447248/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251670 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251663.146786:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251663.146788:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adbb0 00000400:00000200:5.0:1547251663.146789:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251663.146790:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251663.146793:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37934b00 x1622384012447248/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251670 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251663.146795:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251663.146798:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37934b00 x1622384012447248/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251670 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251663.146802:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00010000:00080000:6.0:1547251663.146805:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37934b00 x1622384012447248/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251670 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251665.162686:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447264, portal 4 00000100:00000200:6.0:1547251665.162688:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447296, portal 4 00000100:00000200:10.0:1547251665.162689:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447264, offset 0 00000100:00000200:6.0:1547251665.162690:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447296, offset 0 00000400:00000200:10.0:1547251665.162692:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:6.0:1547251665.162694:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:10.0:1547251665.162695:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:6.0:1547251665.162696:0:12677:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:10.0:1547251665.162697:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23083] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:6.0:1547251665.162699:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23083] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:10.0:1547251665.162700:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23080] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23083] 00000400:00000200:6.0:1547251665.162702:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23080] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23083] 00000400:00000200:10.0:1547251665.162703:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:10.0:1547251665.162704:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251665.162705:0:12677:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:10.0:1547251665.162706:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251665.162706:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251665.162707:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9764, 9764] 00000400:00000200:6.0:1547251665.162708:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251665.162709:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:6.0:1547251665.162709:0:12677:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9764, 9764] 00000400:00000200:6.0:1547251665.162711:0:12677:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:10.0:1547251665.162714:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:10.0:1547251665.162716:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:10.0:1547251665.162719:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000400:00000200:6.0:1547251665.162720:0:12677:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:10.0:1547251665.162721:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:10.0:1547251665.162722:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:6.0:1547251665.162723:0:12677:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:10.0:1547251665.162724:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:2.0:1547251665.162725:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447280, portal 4 00000800:00000200:6.0:1547251665.162726:0:12677:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:6.0:1547251665.162728:0:12677:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:6.0:1547251665.162729:0:12677:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000100:00000200:2.0:1547251665.162729:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447280, offset 0 00000800:00000200:6.0:1547251665.162731:0:12677:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:2.0:1547251665.162733:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:2.0:1547251665.162737:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:2.0:1547251665.162739:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23083] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251665.162742:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:254, d:10, s:23082] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23083] 00000800:00000200:9.2:1547251665.162744:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:2.0:1547251665.162744:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:2.0:1547251665.162746:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251665.162747:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251665.162749:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9765, 9765] 00000400:00000200:2.0:1547251665.162750:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000800:00000200:9.2:1547251665.162753:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:2.0:1547251665.162755:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251665.162758:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:2.0:1547251665.162761:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251665.162763:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:2.0:1547251665.162764:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:2.0:1547251665.162767:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:7.2:1547251665.162783:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000800:00000200:15.0:1547251665.162798:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:15.0:1547251665.162802:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:15.0:1547251665.162805:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960e8f00 x1622384012447264/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251672 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251665.162811:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914dd0 00000400:00000200:15.0:1547251665.162812:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:15.0:1547251665.162814:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251665.162815:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:15.0:1547251665.162817:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251665.162818:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:15.0:1547251665.162819:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:15.0:1547251665.162821:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000100:00000200:12.0:1547251665.162822:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37934e00 x1622384012447296/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251672 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:15.0:1547251665.162823:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b6f00 x1622384012447280/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251672 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251665.162827:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990908 00000400:00000200:15.0:1547251665.162828:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06aaa0 00000400:00000200:12.0:1547251665.162828:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:15.0:1547251665.162829:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251665.162831:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:15.0:1547251665.162832:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:12.0:1547251665.162833:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:15.0:1547251665.162835:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251665.162836:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:15.0:1547251665.162837:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:12.0:1547251665.162838:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:7.2:1547251665.162893:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251665.162904:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:15.0:1547251665.162910:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:15.0:1547251665.162913:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.93@o2ib10 00000800:00000200:7.2:1547251665.162914:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000400:00000200:15.0:1547251665.162916:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:12.0:1547251665.162917:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:12.0:1547251665.162919:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000400:00000200:15.0:1547251665.162920:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04e20 00000400:00000200:15.0:1547251665.162923:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7ff3d [1] + 192 00000400:00000200:12.0:1547251665.162923:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000100:00000200:15.0:1547251665.162927:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960e8f00 x1622384012447264/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251672 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251665.162927:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04e30 00000400:00000200:15.0:1547251665.162937:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251665.162939:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000400:00000200:12.0:1547251665.162939:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7ff5d [1] + 192 00000800:00000200:15.0:1547251665.162941:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:15.0:1547251665.162941:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:15.0:1547251665.162950:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:12.0:1547251665.162950:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b6f00 x1622384012447280/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251672 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251665.162952:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:15.0:1547251665.162954:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251665.162957:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251665.162959:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04e40 00000400:00000200:12.0:1547251665.162961:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251665.162963:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000100:00000200:10.0:1547251665.162964:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960e8f00 x1622384012447264/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251672 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:12.0:1547251665.162965:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:12.0:1547251665.162966:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:12.0:1547251665.162967:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:12.0:1547251665.162968:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:10.0:1547251665.162969:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960e8f00 x1622384012447264/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251672 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251665.162973:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a9145d8 00000400:00000200:15.0:1547251665.162976:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7ff4d [1] + 192 00010000:00080000:10.0:1547251665.162977:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba960e8f00 x1622384012447264/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251672 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:2.0:1547251665.162979:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b6f00 x1622384012447280/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251672 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251665.162985:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b6f00 x1622384012447280/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251672 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251665.162989:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a5d8 00000100:00000200:15.0:1547251665.162991:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37934e00 x1622384012447296/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251672 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:2.0:1547251665.162993:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b6f00 x1622384012447280/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251672 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:15.0:1547251665.162999:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251665.163000:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:15.0:1547251665.163001:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:15.0:1547251665.163002:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000100:00000200:6.0:1547251665.163009:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37934e00 x1622384012447296/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251672 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251665.163016:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37934e00 x1622384012447296/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251672 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251665.163020:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00010000:00080000:6.0:1547251665.163024:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ae37934e00 x1622384012447296/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251672 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:15.0:1547251667.274683:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447312, portal 4 00000100:00000200:15.0:1547251667.274686:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447312, offset 0 00000400:00000200:15.0:1547251667.274690:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:15.0:1547251667.274693:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:15.0:1547251667.274695:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23084] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251667.274698:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23082] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23084] 00000400:00000200:15.0:1547251667.274702:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:15.0:1547251667.274703:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:3.0:1547251667.274704:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447328, portal 4 00000400:00000200:15.0:1547251667.274705:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251667.274706:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9770, 9770] 00000100:00000200:3.0:1547251667.274707:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447328, offset 0 00000400:00000200:15.0:1547251667.274708:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:3.0:1547251667.274710:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:15.0:1547251667.274713:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000400:00000200:3.0:1547251667.274713:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:15.0:1547251667.274716:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:3.0:1547251667.274716:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23084] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:15.0:1547251667.274719:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000100:00000200:9.0:1547251667.274719:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447344, portal 4 00000400:00000200:3.0:1547251667.274719:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23083] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23084] 00000800:00000200:15.0:1547251667.274721:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:15.0:1547251667.274722:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000100:00000200:9.0:1547251667.274722:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447344, offset 0 00000400:00000200:3.0:1547251667.274722:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251667.274723:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:15.0:1547251667.274725:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:9.0:1547251667.274725:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:3.0:1547251667.274725:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251667.274726:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9770, 9771] 00000400:00000200:9.0:1547251667.274728:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:3.0:1547251667.274728:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:9.0:1547251667.274731:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23085] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251667.274734:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23083] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23085] 00000400:00000200:9.0:1547251667.274736:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251667.274738:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251667.274739:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251667.274740:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9771, 9771] 00000400:00000200:9.0:1547251667.274742:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:9.0:1547251667.274747:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251667.274749:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:3.0:1547251667.274752:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251667.274755:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:9.0:1547251667.274757:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251667.274758:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251667.274759:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:9.0:1547251667.274760:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000800:00000200:3.0:1547251667.274761:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:9.0:1547251667.274763:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:3.0:1547251667.274763:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (23)++ 00000800:00000200:3.0:1547251667.274766:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:7.2:1547251667.274776:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251667.274801:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:11.0:1547251667.274810:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:5.2:1547251667.274812:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (22)++ 00000100:00000200:11.0:1547251667.274815:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8ca8300 x1622384012447312/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251667.274818:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251667.274821:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (23)++ 00000800:00000200:10.0:1547251667.274822:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (24)-- 00000400:00000200:11.0:1547251667.274823:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1cc0 00000400:00000200:11.0:1547251667.274824:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:10.0:1547251667.274826:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf2400 x1622384012447344/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251667.274828:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251667.274831:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000400:00000200:10.0:1547251667.274832:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea4c8 00000800:00000200:11.0:1547251667.274833:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (23)++ 00000800:00000200:11.0:1547251667.274834:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (24)-- 00000400:00000200:10.0:1547251667.274834:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:11.0:1547251667.274837:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c2700 x1622384012447328/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251667.274837:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251667.274839:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251667.274841:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:11.0:1547251667.274843:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555660 00000400:00000200:11.0:1547251667.274844:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251667.274847:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251667.274848:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:5.2:1547251667.274906:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251667.274913:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:10.0:1547251667.274920:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251667.274924:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251667.274926:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251667.274928:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251667.274929:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251667.274933:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251667.274933:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04e70 00000400:00000200:11.0:1547251667.274937:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04e50 00000400:00000200:10.0:1547251667.274938:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd74b9 [1] + 192 00000400:00000200:11.0:1547251667.274942:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7499 [1] + 192 00000100:00000200:10.0:1547251667.274954:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf2400 x1622384012447344/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251667.274966:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251667.274967:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8ca8300 x1622384012447312/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251667.274968:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:10.0:1547251667.274969:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251667.274971:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251667.274972:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251667.274974:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251667.274977:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:10.0:1547251667.274978:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251667.274979:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000100:00000200:9.0:1547251667.274979:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf2400 x1622384012447344/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251667.274981:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:11.0:1547251667.274982:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:10.0:1547251667.274982:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04e60 00000800:00000200:11.0:1547251667.274984:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:11.0:1547251667.274985:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000100:00000200:9.0:1547251667.274985:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf2400 x1622384012447344/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251667.274989:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea3b8 00000400:00000200:10.0:1547251667.274992:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd74a9 [1] + 192 00010000:00080000:9.0:1547251667.274992:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf2400 x1622384012447344/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:15.0:1547251667.274996:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8ca8300 x1622384012447312/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:15.0:1547251667.275002:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8ca8300 x1622384012447312/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251667.275006:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb15d8 00000100:00000200:10.0:1547251667.275009:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c2700 x1622384012447328/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:15.0:1547251667.275010:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8ca8300 x1622384012447312/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251667.275017:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251667.275018:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251667.275020:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251667.275021:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:3.0:1547251667.275026:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c2700 x1622384012447328/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251667.275032:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c2700 x1622384012447328/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251667.275036:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251667.275039:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c2700 x1622384012447328/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:0.2:1547251667.640957:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b81d647a00] (20)++ 00000800:00000200:12.0:1547251667.641014:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:12.0:1547251667.641025:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.10.184@o2ib10 00000400:00000200:12.0:1547251667.641029:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.10.184@o2ib10 : PUT - for me 00000400:00000200:12.0:1547251667.641034:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.10.184@o2ib10 of length 520 into portal 26 MB=0x5c3934d100040 00000400:00000200:12.0:1547251667.641037:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.10.184@o2ib10 of length 520/520 into md 0xd5539 [1] + 0 00000400:00000200:12.0:1547251667.641040:0:12665:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d990 00000100:00000200:12.0:1547251667.641042:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:12.0:1547251667.641048:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d990 00000400:00000200:12.0:1547251667.641049:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:12.0:1547251667.641050:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (22)++ 00000800:00000200:12.0:1547251667.641052:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (23)-- 00000800:00000200:12.0:1547251667.641053:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)-- 00000800:00000200:12.0:1547251667.641055:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (21)-- 00000100:00000200:2.0:1547251667.641082:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622412304056384 00010000:02000400:2.0:1547251667.641092:0:12929:0:(ldlm_lib.c:1152:target_handle_connect()) MGS: Received new LWP connection from 10.0.10.184@o2ib10, removing former export from same NID 00010000:00080000:2.0:1547251667.641098:0:12929:0:(ldlm_lib.c:1228:target_handle_connect()) MGS: connection from 8534aa65-7ac8-198c-0718-3e6f83d57705@10.0.10.184@o2ib10 t0 exp ffff97b9b3928400 cur 1547251667 last 1547251642 00000400:00000200:2.0:1547251667.641106:0:12929:0:(peer.c:1106:LNetPrimaryNID()) NID 10.0.10.184@o2ib10 primary NID 10.0.10.184@o2ib10 rc 0 00000100:00080000:2.0:1547251667.641108:0:12929:0:(import.c:104:ptlrpc_import_enter_resend()) ffff97b824577000 : changing import state from FULL to RECOVER 00000100:00080000:2.0:1547251667.641112:0:12929:0:(import.c:1528:ptlrpc_import_recovery_state_machine()) ffff97b824577000 : changing import state from RECOVER to FULL 00000100:02000000:2.0:1547251667.641114:0:12929:0:(import.c:1534:ptlrpc_import_recovery_state_machine()) MGS: Connection restored to f7adbd22-dd00-137c-616b-9c8e12a50a01 (at 10.0.10.184@o2ib10) 00010000:00000200:2.0:1547251667.641121:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692feb450 x1622412304056384/t0(0) o250->8534aa65-7ac8-198c-0718-3e6f83d57705@10.0.10.184@o2ib10:0/0 lens 520/416 e 0 to 0 dl 1547251767 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:2.0:1547251667.641127:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 25, xid 1622412304056384, offset 0 00000400:00000200:2.0:1547251667.641130:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.10.184@o2ib10 00000400:00000200:2.0:1547251667.641132:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.10.184@o2ib10 local destination 00000400:00000200:2.0:1547251667.641135:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251667.641136:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251667.641137:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.0.10.184@o2ib10 c:[8, 8], s:[17, 18] 00000400:00000200:2.0:1547251667.641139:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.10.184@o2ib10 00000400:00000200:2.0:1547251667.641144:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.10.184@o2ib10(10.0.10.184@o2ib10:10.0.10.184@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251667.641147:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 384 bytes in 1 frags to 12345-10.0.10.184@o2ib10 00000800:00000200:2.0:1547251667.641150:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b971916c00] -> 10.0.10.184@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251667.641152:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b81d647a00] (20)++ 00000800:00000200:2.0:1547251667.641154:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:2.0:1547251667.641156:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b81d647a00] (22)-- 00000100:00000200:2.0:1547251667.641160:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:0.2:1547251667.641168:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:12.0:1547251667.641180:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)++ 00000800:00000200:12.0:1547251667.641186:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b81d647a00] (23)-- 00000400:00000200:12.0:1547251667.641189:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a5d8 00000400:00000200:12.0:1547251667.641190:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:12.0:1547251667.641192:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.10.184@o2ib10: PUT: OK 00000800:00000200:15.0:1547251667.641194:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)-- 00000800:00000200:12.0:1547251667.641196:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (21)-- 00000800:00000200:8.2:1547251667.641526:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d02b200] (20)++ 00000800:00000200:15.0:1547251667.641565:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b200] (21)++ 00000800:00000200:15.0:1547251667.641574:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.10.184@o2ib10 00000400:00000200:15.0:1547251667.641578:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.10.184@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251667.641582:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.10.184@o2ib10 of length 328 into portal 26 MB=0x5c3934d100030 00000400:00000200:15.0:1547251667.641597:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.10.184@o2ib10 of length 328/328 into md 0xd5579 [1] + 0 00000400:00000200:15.0:1547251667.641600:0:12666:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d330 00000100:00000200:15.0:1547251667.641602:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:15.0:1547251667.641607:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d330 00000400:00000200:15.0:1547251667.641609:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251667.641610:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (22)++ 00000800:00000200:15.0:1547251667.641612:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (23)-- 00000800:00000200:15.0:1547251667.641613:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b200] (22)-- 00000800:00000200:12.0:1547251667.641616:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b200] (21)-- 00000100:00000200:2.0:1547251667.641655:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622412304056368 00010000:00000200:2.0:1547251667.641674:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692feac50 x1622412304056368/t0(0) o101->8534aa65-7ac8-198c-0718-3e6f83d57705@10.0.10.184@o2ib10:733/0 lens 328/344 e 0 to 0 dl 1547251678 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:2.0:1547251667.641680:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 312 bytes to portal 25, xid 1622412304056368, offset 192 00000400:00000200:2.0:1547251667.641682:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.10.184@o2ib10 00000400:00000200:2.0:1547251667.641685:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.10.184@o2ib10 local destination 00000400:00000200:2.0:1547251667.641688:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251667.641689:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251667.641690:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.0.10.184@o2ib10 c:[8, 8], s:[18, 18] 00000400:00000200:2.0:1547251667.641692:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.2.10.184@o2ib10 00000400:00000200:2.0:1547251667.641697:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.2.10.184@o2ib10(10.0.10.184@o2ib10:10.2.10.184@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251667.641699:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 312 bytes in 1 frags to 12345-10.2.10.184@o2ib10 00000800:00000200:2.0:1547251667.641703:0:12929:0:(o2iblnd_cb.c:1604:kiblnd_launch_tx()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (1)++ 00000800:00000200:2.0:1547251667.641706:0:12929:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:2.0:1547251667.641720:0:12929:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:2.0:1547251667.641722:0:12929:0:(o2iblnd_cb.c:1611:kiblnd_launch_tx()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000100:00000200:2.0:1547251667.641725:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:14.0:1547251667.641738:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:14.0:1547251667.641748:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:14.0:1547251667.642311:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:14.0:1547251667.642324:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba8b02e000 0x164b02e000(0x164b02e000) 00000800:00000200:14.0:1547251667.642325:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ac55e79000 0x815e79000(0x815e79000) 00000800:00000200:14.0:1547251667.642326:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97babd1e2000 0x167d1e2000(0x167d1e2000) 00000800:00000200:14.0:1547251667.642327:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97a65d0e5000 0x21d0e5000(0x21d0e5000) 00000800:00000200:14.0:1547251667.642328:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97bae1cb6000 0x16a1cb6000(0x16a1cb6000) 00000800:00000200:14.0:1547251667.642329:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97badb957000 0x169b957000(0x169b957000) 00000800:00000200:14.0:1547251667.642330:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97af8f584000 0xb4f584000(0xb4f584000) 00000800:00000200:14.0:1547251667.642331:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ae6aab3000 0xa2aab3000(0xa2aab3000) 00000800:00000200:14.0:1547251667.642332:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ba9b663000 0x165b663000(0x165b663000) 00000800:00000200:14.0:1547251667.642333:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ae0f84c000 0x9cf84c000(0x9cf84c000) 00000800:00000200:14.0:1547251667.642333:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97baa12ee000 0x16612ee000(0x16612ee000) 00000800:00000200:14.0:1547251667.642334:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97a72db0b000 0x2edb0b000(0x2edb0b000) 00000800:00000200:14.0:1547251667.642335:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba8f7b5000 0x164f7b5000(0x164f7b5000) 00000800:00000200:14.0:1547251667.642336:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97bade7f1000 0x169e7f1000(0x169e7f1000) 00000800:00000200:14.0:1547251667.642337:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b17b1b1000 0xd3b1b1000(0xd3b1b1000) 00000800:00000200:14.0:1547251667.642338:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97badb94f000 0x169b94f000(0x169b94f000) 00000800:00000200:14.0:1547251667.642339:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b158afd000 0xd18afd000(0xd18afd000) 00000800:00000200:14.0:1547251667.642339:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97badc907000 0x169c907000(0x169c907000) 00000800:00000200:9.0:1547251667.643090:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643092:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643093:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643094:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643094:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643095:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643096:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643096:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643097:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643097:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643098:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643099:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643099:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643100:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643100:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643101:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643102:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643102:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643103:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643103:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643104:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643105:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643105:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643106:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643106:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643108:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643108:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643109:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643109:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643110:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643111:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643111:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643112:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643113:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.643113:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.643114:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000100:12.0:1547251667.643399:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251667.643402:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251667.643404:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251667.643439:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:15.0:1547251667.643455:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (20)++ 00000800:00000200:12.0:1547251667.643460:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97babe8ae000] (21)-- 00000800:00000200:15.0:1547251667.643464:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:15.0:1547251667.643465:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (19)-- 00000800:00000200:15.0:1547251667.643467:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (18)-- 00000800:00000200:8.2:1547251667.643581:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ae000] (17)++ 00000800:00000200:15.0:1547251667.643602:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (18)++ 00000800:00000200:15.0:1547251667.643606:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (19)-- 00000800:00000200:15.0:1547251667.643608:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (18)-- 00000800:00000200:15.0:1547251667.643609:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (17)++ 00000800:00000200:15.0:1547251667.643610:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (18)-- 00000800:00000200:15.0:1547251667.643612:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (17)-- 00000800:00000200:13.0:1547251667.643613:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (16)++ 00000800:00000200:13.0:1547251667.643621:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (17)-- 00000800:00000200:13.0:1547251667.643622:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (16)-- 00000800:00000200:13.0:1547251667.643623:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (15)++ 00000800:00000200:13.0:1547251667.643628:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (16)-- 00000800:00000200:13.0:1547251667.643629:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (15)-- 00000800:00000200:13.0:1547251667.643630:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (14)++ 00000800:00000200:13.0:1547251667.643635:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (15)-- 00000800:00000200:13.0:1547251667.643636:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (14)-- 00000800:00000200:13.0:1547251667.643637:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (13)++ 00000800:00000200:13.0:1547251667.643641:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (14)-- 00000800:00000200:13.0:1547251667.643642:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (13)-- 00000800:00000200:13.0:1547251667.643644:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (12)++ 00000800:00000200:13.0:1547251667.643648:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (13)-- 00000800:00000200:13.0:1547251667.643649:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (12)-- 00000800:00000200:13.0:1547251667.643650:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (11)++ 00000800:00000200:13.0:1547251667.643654:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (12)-- 00000800:00000200:13.0:1547251667.643655:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (11)-- 00000800:00000200:13.0:1547251667.643656:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (10)++ 00000800:00000200:13.0:1547251667.643660:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (11)-- 00000800:00000200:13.0:1547251667.643661:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (10)-- 00000800:00000200:13.0:1547251667.643663:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (9)++ 00000800:00000200:13.0:1547251667.643667:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (10)-- 00000800:00000200:13.0:1547251667.643668:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (9)-- 00000800:00000200:13.0:1547251667.643669:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (8)++ 00000800:00000200:13.0:1547251667.643673:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (9)-- 00000800:00000200:13.0:1547251667.643674:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (8)-- 00000800:00000200:13.0:1547251667.643676:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (7)++ 00000800:00000200:13.0:1547251667.643680:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (8)-- 00000800:00000200:13.0:1547251667.643681:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (7)-- 00000800:00000200:13.0:1547251667.643682:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (6)++ 00000800:00000200:13.0:1547251667.643688:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (7)-- 00000800:00000200:13.0:1547251667.643688:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (6)-- 00000800:00000200:13.0:1547251667.643689:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (5)++ 00000800:00000200:13.0:1547251667.643692:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (6)-- 00000800:00000200:13.0:1547251667.643693:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (5)-- 00000800:00000200:13.0:1547251667.643694:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (4)++ 00000800:00000200:15.0:1547251667.643695:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (5)++ 00000800:00000200:13.0:1547251667.643695:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (5)-- 00000800:00000200:13.0:1547251667.643696:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (5)-- 00000800:00000200:15.0:1547251667.643697:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (4)-- 00000800:00000200:13.0:1547251667.643697:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (4)++ 00000800:00000200:15.0:1547251667.643698:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (4)-- 00000800:00000200:13.0:1547251667.643699:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (3)-- 00000800:00000200:15.0:1547251667.643700:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (3)-- 00000800:00000200:13.0:1547251667.643701:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (1)-- 00000800:00000200:9.0:1547251667.643709:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.644459:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251667.644464:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.644475:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251667.644477:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:14.0:1547251667.644483:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:14.0:1547251667.644493:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:14.0:1547251667.645017:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:14.0:1547251667.645025:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba9ea1b000 0x165ea1b000(0x165ea1b000) 00000800:00000200:14.0:1547251667.645027:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97babe286000 0x167e286000(0x167e286000) 00000800:00000200:14.0:1547251667.645028:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97baa1673000 0x1661673000(0x1661673000) 00000800:00000200:14.0:1547251667.645029:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97a6c4c9d000 0x284c9d000(0x284c9d000) 00000800:00000200:14.0:1547251667.645030:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97baa1e8b000 0x1661e8b000(0x1661e8b000) 00000800:00000200:14.0:1547251667.645031:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97baa2b25000 0x1662b25000(0x1662b25000) 00000800:00000200:14.0:1547251667.645032:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97baa209f000 0x166209f000(0x166209f000) 00000800:00000200:14.0:1547251667.645033:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97a65d2ff000 0x21d2ff000(0x21d2ff000) 00000800:00000200:14.0:1547251667.645034:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ad5727d000 0x91727d000(0x91727d000) 00000800:00000200:14.0:1547251667.645035:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b3b5c04000 0xf75c04000(0xf75c04000) 00000800:00000200:14.0:1547251667.645036:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ba9ae84000 0x165ae84000(0x165ae84000) 00000800:00000200:14.0:1547251667.645037:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b1a46a7000 0xd646a7000(0xd646a7000) 00000800:00000200:14.0:1547251667.645038:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b81d6e9000 0x13dd6e9000(0x13dd6e9000) 00000800:00000200:14.0:1547251667.645039:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ae379f2000 0x9f79f2000(0x9f79f2000) 00000800:00000200:14.0:1547251667.645040:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b13395b000 0xcf395b000(0xcf395b000) 00000800:00000200:14.0:1547251667.645041:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b8246e3000 0x13e46e3000(0x13e46e3000) 00000800:00000200:14.0:1547251667.645041:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ae6aa3c000 0xa2aa3c000(0xa2aa3c000) 00000800:00000200:14.0:1547251667.645042:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ae6aaa5000 0xa2aaa5000(0xa2aaa5000) 00000800:00000200:9.0:1547251667.645692:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645694:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645694:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645695:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645696:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645696:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645697:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645698:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645698:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645699:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645699:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645700:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645701:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645701:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645702:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645703:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645703:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645704:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645704:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645705:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645706:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645707:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645708:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645708:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645709:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645709:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645710:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645711:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645711:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645712:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645712:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645713:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645714:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645714:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:9.0:1547251667.645715:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:9.0:1547251667.645715:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000100:12.0:1547251667.645992:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251667.645995:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251667.645997:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251667.646034:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:12.0:1547251667.646049:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:15.0:1547251667.646081:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (19)++ 00000800:00000200:15.0:1547251667.646091:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (20)-- 00000800:00000200:15.0:1547251667.646092:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (19)-- 00000800:00000200:15.0:1547251667.646093:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (18)-- 00000800:00000200:8.2:1547251667.646175:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ae000] (17)++ 00000800:00000200:15.0:1547251667.646186:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (18)++ 00000800:00000200:15.0:1547251667.646190:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (19)-- 00000800:00000200:15.0:1547251667.646191:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (18)-- 00000800:00000200:15.0:1547251667.646192:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (17)++ 00000800:00000200:15.0:1547251667.646193:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (18)-- 00000800:00000200:15.0:1547251667.646194:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (17)-- 00000800:00000200:15.0:1547251667.646194:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (16)++ 00000800:00000200:15.0:1547251667.646195:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (17)-- 00000800:00000200:15.0:1547251667.646196:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (16)-- 00000800:00000200:15.0:1547251667.646197:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (15)++ 00000800:00000200:15.0:1547251667.646200:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (16)-- 00000800:00000200:15.0:1547251667.646201:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (15)-- 00000800:00000200:15.0:1547251667.646202:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (14)++ 00000800:00000200:15.0:1547251667.646203:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (15)-- 00000800:00000200:15.0:1547251667.646204:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (14)-- 00000800:00000200:13.0:1547251667.646205:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (13)++ 00000800:00000200:13.0:1547251667.646210:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (14)-- 00000800:00000200:13.0:1547251667.646211:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (13)-- 00000800:00000200:13.0:1547251667.646212:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (12)++ 00000800:00000200:13.0:1547251667.646215:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (13)-- 00000800:00000200:13.0:1547251667.646216:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (12)-- 00000800:00000200:13.0:1547251667.646217:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (11)++ 00000800:00000200:15.0:1547251667.646218:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (12)++ 00000800:00000200:13.0:1547251667.646218:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (12)-- 00000800:00000200:13.0:1547251667.646219:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (12)-- 00000800:00000200:15.0:1547251667.646220:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (11)-- 00000800:00000200:13.0:1547251667.646220:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (11)++ 00000800:00000200:15.0:1547251667.646221:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (11)-- 00000800:00000200:15.0:1547251667.646222:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (10)++ 00000800:00000200:13.0:1547251667.646222:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (10)-- 00000800:00000200:13.0:1547251667.646225:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (10)-- 00000800:00000200:15.0:1547251667.646226:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (9)-- 00000800:00000200:13.0:1547251667.646226:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (9)++ 00000800:00000200:15.0:1547251667.646227:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (9)-- 00000800:00000200:15.0:1547251667.646228:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (8)++ 00000800:00000200:13.0:1547251667.646228:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (8)-- 00000800:00000200:13.0:1547251667.646229:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (8)-- 00000800:00000200:15.0:1547251667.646230:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (7)-- 00000800:00000200:13.0:1547251667.646230:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (7)++ 00000800:00000200:15.0:1547251667.646231:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (7)-- 00000800:00000200:15.0:1547251667.646232:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (6)++ 00000800:00000200:13.0:1547251667.646232:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (6)-- 00000800:00000200:13.0:1547251667.646233:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (6)-- 00000800:00000200:15.0:1547251667.646234:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (5)-- 00000800:00000200:13.0:1547251667.646234:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (5)++ 00000800:00000200:15.0:1547251667.646235:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (5)-- 00000800:00000200:15.0:1547251667.646236:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae000] (4)++ 00000800:00000200:13.0:1547251667.646236:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (4)-- 00000800:00000200:13.0:1547251667.646237:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (4)-- 00000800:00000200:15.0:1547251667.646238:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae000] (3)-- 00000800:00000200:13.0:1547251667.646238:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (3)-- 00000800:00000200:15.0:1547251667.646239:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae000] (1)-- 00000800:00000200:9.0:1547251667.646251:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.646982:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251667.646986:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.646995:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251667.646996:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251667.647003:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251667.647012:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251667.647487:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251667.647496:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97bae1a95000 0x16a1a95000(0x16a1a95000) 00000800:00000200:10.0:1547251667.647497:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba912e1000 0x16512e1000(0x16512e1000) 00000800:00000200:10.0:1547251667.647498:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b81d44f000 0x13dd44f000(0x13dd44f000) 00000800:00000200:10.0:1547251667.647499:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97babd50e000 0x167d50e000(0x167d50e000) 00000800:00000200:10.0:1547251667.647500:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ba9ffcc000 0x165ffcc000(0x165ffcc000) 00000800:00000200:10.0:1547251667.647501:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba9d3ea000 0x165d3ea000(0x165d3ea000) 00000800:00000200:10.0:1547251667.647502:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97baa0e56000 0x1660e56000(0x1660e56000) 00000800:00000200:10.0:1547251667.647503:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97baa0e9e000 0x1660e9e000(0x1660e9e000) 00000800:00000200:10.0:1547251667.647504:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97bae273e000 0x16a273e000(0x16a273e000) 00000800:00000200:10.0:1547251667.647504:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97bae1bf4000 0x16a1bf4000(0x16a1bf4000) 00000800:00000200:10.0:1547251667.647505:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97bade64f000 0x169e64f000(0x169e64f000) 00000800:00000200:10.0:1547251667.647506:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba927fa000 0x16527fa000(0x16527fa000) 00000800:00000200:10.0:1547251667.647507:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba9a6df000 0x165a6df000(0x165a6df000) 00000800:00000200:10.0:1547251667.647508:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b17b056000 0xd3b056000(0xd3b056000) 00000800:00000200:10.0:1547251667.647509:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b5bdd35000 0x117dd35000(0x117dd35000) 00000800:00000200:10.0:1547251667.647510:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97bae1342000 0x16a1342000(0x16a1342000) 00000800:00000200:10.0:1547251667.647511:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97a8c76a9000 0x4876a9000(0x4876a9000) 00000800:00000200:10.0:1547251667.647512:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ae37822000 0x9f7822000(0x9f7822000) 00000800:00000200:9.0:1547251667.648119:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648120:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648121:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648122:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648122:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648123:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648123:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648124:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648125:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648125:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648126:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648127:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648127:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648128:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648128:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648129:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648130:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648131:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648131:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648132:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648133:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648133:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648134:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648134:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648135:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648136:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648136:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648137:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648137:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648138:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648138:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648139:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648140:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648140:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251667.648141:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251667.648141:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000100:12.0:1547251667.648465:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251667.648468:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251667.648470:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251667.648507:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:13.0:1547251667.648528:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (20)++ 00000800:00000200:13.0:1547251667.648536:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (21)-- 00000800:00000200:13.0:1547251667.648536:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:13.0:1547251667.648538:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (19)-- 00000800:00000200:12.0:1547251667.648553:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c200] (18)-- 00000800:00000200:8.2:1547251667.648606:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c200] (17)++ 00000800:00000200:13.0:1547251667.648621:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (18)++ 00000800:00000200:13.0:1547251667.648627:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (19)-- 00000800:00000200:13.0:1547251667.648628:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (18)-- 00000800:00000200:13.0:1547251667.648630:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (17)++ 00000800:00000200:13.0:1547251667.648631:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (18)-- 00000800:00000200:13.0:1547251667.648632:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (17)-- 00000800:00000200:13.0:1547251667.648633:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (16)++ 00000800:00000200:13.0:1547251667.648635:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (17)-- 00000800:00000200:13.0:1547251667.648636:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (16)-- 00000800:00000200:15.0:1547251667.648638:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (15)++ 00000800:00000200:15.0:1547251667.648644:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (16)-- 00000800:00000200:15.0:1547251667.648645:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (15)-- 00000800:00000200:15.0:1547251667.648647:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (14)++ 00000800:00000200:15.0:1547251667.648650:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (15)-- 00000800:00000200:15.0:1547251667.648650:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (14)-- 00000800:00000200:15.0:1547251667.648651:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (13)++ 00000800:00000200:15.0:1547251667.648652:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (14)-- 00000800:00000200:15.0:1547251667.648653:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (14)-- 00000800:00000200:13.0:1547251667.648653:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (14)++ 00000800:00000200:13.0:1547251667.648654:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (13)-- 00000800:00000200:15.0:1547251667.648655:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (13)++ 00000800:00000200:13.0:1547251667.648656:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (13)-- 00000800:00000200:15.0:1547251667.648657:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (12)-- 00000800:00000200:13.0:1547251667.648657:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (12)++ 00000800:00000200:15.0:1547251667.648658:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (12)-- 00000800:00000200:15.0:1547251667.648659:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (11)++ 00000800:00000200:13.0:1547251667.648659:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (11)-- 00000800:00000200:13.0:1547251667.648660:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (11)-- 00000800:00000200:15.0:1547251667.648661:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (10)-- 00000800:00000200:13.0:1547251667.648661:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (10)++ 00000800:00000200:15.0:1547251667.648662:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (10)-- 00000800:00000200:15.0:1547251667.648663:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (9)++ 00000800:00000200:13.0:1547251667.648663:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (9)-- 00000800:00000200:13.0:1547251667.648664:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (9)-- 00000800:00000200:15.0:1547251667.648665:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (8)-- 00000800:00000200:13.0:1547251667.648665:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (8)++ 00000800:00000200:15.0:1547251667.648666:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (8)-- 00000800:00000200:15.0:1547251667.648667:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (7)++ 00000800:00000200:13.0:1547251667.648667:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (7)-- 00000800:00000200:13.0:1547251667.648668:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (7)-- 00000800:00000200:15.0:1547251667.648669:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (6)-- 00000800:00000200:13.0:1547251667.648669:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (6)++ 00000800:00000200:15.0:1547251667.648670:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (6)-- 00000800:00000200:13.0:1547251667.648670:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (5)-- 00000800:00000200:15.0:1547251667.648671:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (5)++ 00000800:00000200:15.0:1547251667.648672:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (4)-- 00000800:00000200:13.0:1547251667.648672:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (5)-- 00000800:00000200:15.0:1547251667.648673:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (4)-- 00000800:00000200:13.0:1547251667.648673:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (4)++ 00000800:00000200:13.0:1547251667.648674:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (3)-- 00000800:00000200:15.0:1547251667.648675:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (3)-- 00000800:00000200:13.0:1547251667.648676:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (1)-- 00000800:00000200:9.0:1547251667.648688:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.649541:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251667.649546:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.649555:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251667.649557:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251667.649563:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251667.649570:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251667.649928:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251667.649935:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97baddebc000 0x169debc000(0x169debc000) 00000800:00000200:10.0:1547251667.649937:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba9ba4b000 0x165ba4b000(0x165ba4b000) 00000800:00000200:10.0:1547251667.649938:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba9bf9c000 0x165bf9c000(0x165bf9c000) 00000800:00000200:10.0:1547251667.649939:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97badd522000 0x169d522000(0x169d522000) 00000800:00000200:10.0:1547251667.649939:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97bad6a8c000 0x1696a8c000(0x1696a8c000) 00000800:00000200:10.0:1547251667.649940:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba9ce49000 0x165ce49000(0x165ce49000) 00000800:00000200:10.0:1547251667.649942:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97bae1cfd000 0x16a1cfd000(0x16a1cfd000) 00000800:00000200:10.0:1547251667.649950:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba90fac000 0x1650fac000(0x1650fac000) 00000800:00000200:10.0:1547251667.649951:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97a601aa3000 0x1c1aa3000(0x1c1aa3000) 00000800:00000200:10.0:1547251667.649952:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba95c64000 0x1655c64000(0x1655c64000) 00000800:00000200:10.0:1547251667.649953:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97bade77b000 0x169e77b000(0x169e77b000) 00000800:00000200:10.0:1547251667.649954:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba957c3000 0x16557c3000(0x16557c3000) 00000800:00000200:10.0:1547251667.649954:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b1231bf000 0xce31bf000(0xce31bf000) 00000800:00000200:10.0:1547251667.649955:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba95ba2000 0x1655ba2000(0x1655ba2000) 00000800:00000200:10.0:1547251667.649956:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba99fe5000 0x1659fe5000(0x1659fe5000) 00000800:00000200:10.0:1547251667.649957:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba94159000 0x1654159000(0x1654159000) 00000800:00000200:10.0:1547251667.649958:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97baa13a5000 0x16613a5000(0x16613a5000) 00000800:00000200:10.0:1547251667.649959:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ae5c43e000 0xa1c43e000(0xa1c43e000) 00000800:00000200:9.0:1547251667.650551:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650552:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650553:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650553:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650554:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650555:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650555:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650556:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650557:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650557:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650558:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650558:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650559:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650560:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650561:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650561:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650562:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650563:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650563:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650564:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650564:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650565:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650566:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650566:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650567:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650567:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650568:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650569:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650569:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650570:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650570:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650571:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650572:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650572:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:9.0:1547251667.650573:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:9.0:1547251667.650574:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000100:12.0:1547251667.650851:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251667.650855:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251667.650856:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251667.650898:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:12.0:1547251667.650908:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:15.0:1547251667.650958:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (19)++ 00000800:00000200:15.0:1547251667.650968:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (20)-- 00000800:00000200:15.0:1547251667.650969:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (19)-- 00000800:00000200:15.0:1547251667.650970:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (18)-- 00000800:00000200:8.2:1547251667.651041:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07ea00] (17)++ 00000800:00000200:15.0:1547251667.651054:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (18)++ 00000800:00000200:15.0:1547251667.651058:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (19)-- 00000800:00000200:15.0:1547251667.651059:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (18)-- 00000800:00000200:15.0:1547251667.651060:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (17)++ 00000800:00000200:15.0:1547251667.651061:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (18)-- 00000800:00000200:15.0:1547251667.651061:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (17)-- 00000800:00000200:15.0:1547251667.651062:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (16)++ 00000800:00000200:15.0:1547251667.651063:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (17)-- 00000800:00000200:15.0:1547251667.651064:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (16)-- 00000800:00000200:15.0:1547251667.651065:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (15)++ 00000800:00000200:15.0:1547251667.651068:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (16)-- 00000800:00000200:15.0:1547251667.651069:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (15)-- 00000800:00000200:15.0:1547251667.651069:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (14)++ 00000800:00000200:15.0:1547251667.651071:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (15)-- 00000800:00000200:15.0:1547251667.651071:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (14)-- 00000800:00000200:13.0:1547251667.651072:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (13)++ 00000800:00000200:13.0:1547251667.651078:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (14)-- 00000800:00000200:13.0:1547251667.651079:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (13)-- 00000800:00000200:13.0:1547251667.651080:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (12)++ 00000800:00000200:13.0:1547251667.651083:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (13)-- 00000800:00000200:13.0:1547251667.651084:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (12)-- 00000800:00000200:13.0:1547251667.651084:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (11)++ 00000800:00000200:15.0:1547251667.651086:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (12)++ 00000800:00000200:13.0:1547251667.651086:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (12)-- 00000800:00000200:13.0:1547251667.651087:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (12)-- 00000800:00000200:15.0:1547251667.651088:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (11)-- 00000800:00000200:13.0:1547251667.651088:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (11)++ 00000800:00000200:15.0:1547251667.651090:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (11)-- 00000800:00000200:15.0:1547251667.651091:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (10)++ 00000800:00000200:13.0:1547251667.651091:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (10)-- 00000800:00000200:13.0:1547251667.651092:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (10)-- 00000800:00000200:15.0:1547251667.651093:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (9)-- 00000800:00000200:13.0:1547251667.651093:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (9)++ 00000800:00000200:15.0:1547251667.651094:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (9)-- 00000800:00000200:15.0:1547251667.651095:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (8)++ 00000800:00000200:13.0:1547251667.651095:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (8)-- 00000800:00000200:13.0:1547251667.651096:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (8)-- 00000800:00000200:15.0:1547251667.651097:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (7)-- 00000800:00000200:13.0:1547251667.651097:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (7)++ 00000800:00000200:15.0:1547251667.651098:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (7)-- 00000800:00000200:15.0:1547251667.651099:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (6)++ 00000800:00000200:13.0:1547251667.651099:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (6)-- 00000800:00000200:13.0:1547251667.651100:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (6)-- 00000800:00000200:15.0:1547251667.651101:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (5)-- 00000800:00000200:13.0:1547251667.651101:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (5)++ 00000800:00000200:15.0:1547251667.651102:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (5)-- 00000800:00000200:15.0:1547251667.651103:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ea00] (4)++ 00000800:00000200:13.0:1547251667.651103:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (4)-- 00000800:00000200:13.0:1547251667.651104:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (4)-- 00000800:00000200:15.0:1547251667.651105:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ea00] (3)-- 00000800:00000200:13.0:1547251667.651105:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (3)-- 00000800:00000200:15.0:1547251667.651106:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ea00] (1)-- 00000800:00000200:9.0:1547251667.651118:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.651898:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251667.651903:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.651912:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251667.651914:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251667.651920:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251667.651928:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251667.652224:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251667.652231:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97a765da6000 0x325da6000(0x325da6000) 00000800:00000200:10.0:1547251667.652232:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97babd481000 0x167d481000(0x167d481000) 00000800:00000200:10.0:1547251667.652233:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97bade7f9000 0x169e7f9000(0x169e7f9000) 00000800:00000200:10.0:1547251667.652234:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b6268ce000 0x11e68ce000(0x11e68ce000) 00000800:00000200:10.0:1547251667.652235:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97badd67c000 0x169d67c000(0x169d67c000) 00000800:00000200:10.0:1547251667.652236:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba9bae9000 0x165bae9000(0x165bae9000) 00000800:00000200:10.0:1547251667.652237:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b6268c2000 0x11e68c2000(0x11e68c2000) 00000800:00000200:10.0:1547251667.652238:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ad5727d000 0x91727d000(0x91727d000) 00000800:00000200:10.0:1547251667.652238:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b3b5c04000 0xf75c04000(0xf75c04000) 00000800:00000200:10.0:1547251667.652239:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba9ae84000 0x165ae84000(0x165ae84000) 00000800:00000200:10.0:1547251667.652240:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ba8b02e000 0x164b02e000(0x164b02e000) 00000800:00000200:10.0:1547251667.652241:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ac55e79000 0x815e79000(0x815e79000) 00000800:00000200:10.0:1547251667.652242:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97babd1e2000 0x167d1e2000(0x167d1e2000) 00000800:00000200:10.0:1547251667.652243:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97a65d0e5000 0x21d0e5000(0x21d0e5000) 00000800:00000200:10.0:1547251667.652244:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97bae1cb6000 0x16a1cb6000(0x16a1cb6000) 00000800:00000200:10.0:1547251667.652244:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97badb957000 0x169b957000(0x169b957000) 00000800:00000200:10.0:1547251667.652246:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97af8f584000 0xb4f584000(0xb4f584000) 00000800:00000200:10.0:1547251667.652246:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ae6aab3000 0xa2aab3000(0xa2aab3000) 00000800:00000200:9.0:1547251667.652850:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652851:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652852:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652852:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652853:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652853:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652854:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652855:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652855:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652857:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652857:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652858:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652858:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652859:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652860:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652860:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652861:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652861:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652862:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652863:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652863:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652864:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652864:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652865:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652866:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652866:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652867:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652867:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652868:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652869:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652869:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652870:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652870:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652871:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251667.652872:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251667.652872:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000100:12.0:1547251667.653155:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251667.653158:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251667.653160:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251667.653195:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:12.0:1547251667.653213:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:13.0:1547251667.653221:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:13.0:1547251667.653229:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:13.0:1547251667.653231:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (19)-- 00000800:00000200:13.0:1547251667.653232:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (18)-- 00000800:00000200:8.2:1547251667.653348:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e000] (17)++ 00000800:00000200:13.0:1547251667.653362:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (18)++ 00000800:00000200:13.0:1547251667.653367:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (19)-- 00000800:00000200:13.0:1547251667.653367:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (18)-- 00000800:00000200:13.0:1547251667.653368:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (17)++ 00000800:00000200:13.0:1547251667.653369:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (18)-- 00000800:00000200:13.0:1547251667.653370:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (17)-- 00000800:00000200:13.0:1547251667.653370:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (16)++ 00000800:00000200:13.0:1547251667.653371:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (17)-- 00000800:00000200:13.0:1547251667.653372:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (16)-- 00000800:00000200:13.0:1547251667.653372:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (15)++ 00000800:00000200:13.0:1547251667.653374:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (16)-- 00000800:00000200:13.0:1547251667.653374:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (15)-- 00000800:00000200:15.0:1547251667.653376:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (14)++ 00000800:00000200:15.0:1547251667.653383:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (15)-- 00000800:00000200:15.0:1547251667.653384:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (14)-- 00000800:00000200:15.0:1547251667.653385:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (13)++ 00000800:00000200:15.0:1547251667.653390:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (14)-- 00000800:00000200:15.0:1547251667.653390:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (13)-- 00000800:00000200:15.0:1547251667.653391:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (12)++ 00000800:00000200:15.0:1547251667.653393:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (13)-- 00000800:00000200:13.0:1547251667.653393:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (13)++ 00000800:00000200:15.0:1547251667.653394:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (13)-- 00000800:00000200:15.0:1547251667.653395:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (12)++ 00000800:00000200:13.0:1547251667.653395:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (12)-- 00000800:00000200:13.0:1547251667.653396:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (12)-- 00000800:00000200:15.0:1547251667.653397:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (11)-- 00000800:00000200:13.0:1547251667.653397:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (11)++ 00000800:00000200:15.0:1547251667.653398:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (11)-- 00000800:00000200:15.0:1547251667.653399:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (10)++ 00000800:00000200:13.0:1547251667.653399:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (10)-- 00000800:00000200:13.0:1547251667.653400:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (10)-- 00000800:00000200:15.0:1547251667.653401:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (9)-- 00000800:00000200:13.0:1547251667.653401:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (9)++ 00000800:00000200:15.0:1547251667.653402:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (9)-- 00000800:00000200:15.0:1547251667.653403:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (8)++ 00000800:00000200:13.0:1547251667.653403:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (8)-- 00000800:00000200:13.0:1547251667.653404:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (8)-- 00000800:00000200:15.0:1547251667.653408:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (7)-- 00000800:00000200:13.0:1547251667.653408:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (7)++ 00000800:00000200:15.0:1547251667.653409:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (7)-- 00000800:00000200:15.0:1547251667.653410:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (6)++ 00000800:00000200:13.0:1547251667.653410:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (6)-- 00000800:00000200:13.0:1547251667.653411:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (6)-- 00000800:00000200:15.0:1547251667.653412:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (5)-- 00000800:00000200:13.0:1547251667.653412:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (5)++ 00000800:00000200:15.0:1547251667.653413:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (5)-- 00000800:00000200:15.0:1547251667.653414:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (4)++ 00000800:00000200:13.0:1547251667.653414:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (4)-- 00000800:00000200:13.0:1547251667.653415:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (4)-- 00000800:00000200:15.0:1547251667.653416:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (3)-- 00000800:00000200:13.0:1547251667.653416:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (3)-- 00000800:00000200:15.0:1547251667.653417:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (1)-- 00000800:00000200:9.0:1547251667.653429:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.654345:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251667.654349:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.654358:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251667.654360:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251667.654366:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251667.654375:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251667.654644:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251667.654651:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97baa12ee000 0x16612ee000(0x16612ee000) 00000800:00000200:10.0:1547251667.654652:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97a72db0b000 0x2edb0b000(0x2edb0b000) 00000800:00000200:10.0:1547251667.654653:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba8f7b5000 0x164f7b5000(0x164f7b5000) 00000800:00000200:10.0:1547251667.654654:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97bade7f1000 0x169e7f1000(0x169e7f1000) 00000800:00000200:10.0:1547251667.654655:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97badb94f000 0x169b94f000(0x169b94f000) 00000800:00000200:10.0:1547251667.654656:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b158afd000 0xd18afd000(0xd18afd000) 00000800:00000200:10.0:1547251667.654657:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97badc907000 0x169c907000(0x169c907000) 00000800:00000200:10.0:1547251667.654658:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b073a9d000 0xc33a9d000(0xc33a9d000) 00000800:00000200:10.0:1547251667.654659:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b6fde3e000 0x12bde3e000(0x12bde3e000) 00000800:00000200:10.0:1547251667.654659:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba92fc4000 0x1652fc4000(0x1652fc4000) 00000800:00000200:10.0:1547251667.654660:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97badc9ea000 0x169c9ea000(0x169c9ea000) 00000800:00000200:10.0:1547251667.654661:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b9073ab000 0x14c73ab000(0x14c73ab000) 00000800:00000200:10.0:1547251667.654662:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba95590000 0x1655590000(0x1655590000) 00000800:00000200:10.0:1547251667.654663:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba92952000 0x1652952000(0x1652952000) 00000800:00000200:10.0:1547251667.654664:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba927e7000 0x16527e7000(0x16527e7000) 00000800:00000200:10.0:1547251667.654665:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba8f6ca000 0x164f6ca000(0x164f6ca000) 00000800:00000200:10.0:1547251667.654665:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97badcaf8000 0x169caf8000(0x169caf8000) 00000800:00000200:10.0:1547251667.654666:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba9b621000 0x165b621000(0x165b621000) 00000800:00000200:9.0:1547251667.655221:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655222:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655223:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655223:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655224:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655225:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655226:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655226:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655227:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655228:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655228:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655229:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655230:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655230:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655231:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655232:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655232:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655233:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655233:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655234:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655235:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655235:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655236:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655237:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655237:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655238:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655238:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655239:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655240:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655240:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655241:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655241:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655242:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655243:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251667.655243:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251667.655244:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000100:12.0:1547251667.655509:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251667.655512:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251667.655514:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251667.655556:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:12.0:1547251667.655573:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:13.0:1547251667.655576:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:13.0:1547251667.655582:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:13.0:1547251667.655602:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (19)-- 00000800:00000200:13.0:1547251667.655604:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (18)-- 00000800:00000200:8.2:1547251667.655697:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07d000] (17)++ 00000800:00000200:13.0:1547251667.655709:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (18)++ 00000800:00000200:13.0:1547251667.655713:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (19)-- 00000800:00000200:13.0:1547251667.655714:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (18)-- 00000800:00000200:13.0:1547251667.655715:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (17)++ 00000800:00000200:13.0:1547251667.655716:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (18)-- 00000800:00000200:13.0:1547251667.655716:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (17)-- 00000800:00000200:13.0:1547251667.655717:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (16)++ 00000800:00000200:13.0:1547251667.655718:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (17)-- 00000800:00000200:13.0:1547251667.655719:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (16)-- 00000800:00000200:13.0:1547251667.655719:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (15)++ 00000800:00000200:13.0:1547251667.655721:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (16)-- 00000800:00000200:13.0:1547251667.655721:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (15)-- 00000800:00000200:15.0:1547251667.655723:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (14)++ 00000800:00000200:15.0:1547251667.655730:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (15)-- 00000800:00000200:15.0:1547251667.655731:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (14)-- 00000800:00000200:15.0:1547251667.655732:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (13)++ 00000800:00000200:15.0:1547251667.655735:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (14)-- 00000800:00000200:15.0:1547251667.655736:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (13)-- 00000800:00000200:15.0:1547251667.655736:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (12)++ 00000800:00000200:15.0:1547251667.655738:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (13)-- 00000800:00000200:13.0:1547251667.655738:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (13)++ 00000800:00000200:15.0:1547251667.655739:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (13)-- 00000800:00000200:15.0:1547251667.655740:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (12)++ 00000800:00000200:13.0:1547251667.655740:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (12)-- 00000800:00000200:13.0:1547251667.655741:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (12)-- 00000800:00000200:15.0:1547251667.655742:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (11)-- 00000800:00000200:13.0:1547251667.655742:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (11)++ 00000800:00000200:15.0:1547251667.655743:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (11)-- 00000800:00000200:15.0:1547251667.655744:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (10)++ 00000800:00000200:13.0:1547251667.655744:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (10)-- 00000800:00000200:13.0:1547251667.655745:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (10)-- 00000800:00000200:15.0:1547251667.655746:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (9)-- 00000800:00000200:13.0:1547251667.655746:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (9)++ 00000800:00000200:15.0:1547251667.655747:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (9)-- 00000800:00000200:15.0:1547251667.655748:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (8)++ 00000800:00000200:13.0:1547251667.655748:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (8)-- 00000800:00000200:13.0:1547251667.655749:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (8)-- 00000800:00000200:15.0:1547251667.655750:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (7)-- 00000800:00000200:13.0:1547251667.655750:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (7)++ 00000800:00000200:13.0:1547251667.655751:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (8)-- 00000800:00000200:13.0:1547251667.655751:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (7)-- 00000800:00000200:15.0:1547251667.655752:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (5)-- 00000800:00000200:13.0:1547251667.655753:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (4)++ 00000800:00000200:13.0:1547251667.655756:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (5)-- 00000800:00000200:13.0:1547251667.655758:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (4)-- 00000800:00000200:13.0:1547251667.655759:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (3)++ 00000800:00000200:13.0:1547251667.655762:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (4)-- 00000800:00000200:13.0:1547251667.655763:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (3)-- 00000800:00000200:13.0:1547251667.655763:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (2)++ 00000800:00000200:13.0:1547251667.655764:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (3)-- 00000800:00000200:13.0:1547251667.655765:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (2)-- 00000800:00000200:15.0:1547251667.655766:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (1)-- 00000800:00000200:9.0:1547251667.655778:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.656633:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251667.656637:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.656647:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251667.656648:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251667.656657:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251667.656664:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251667.656858:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251667.656864:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97badc8d2000 0x169c8d2000(0x169c8d2000) 00000800:00000200:10.0:1547251667.656865:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba92997000 0x1652997000(0x1652997000) 00000800:00000200:10.0:1547251667.656866:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97bae13b0000 0x16a13b0000(0x16a13b0000) 00000800:00000200:10.0:1547251667.656866:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ba92af3000 0x1652af3000(0x1652af3000) 00000800:00000200:10.0:1547251667.656867:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97bae04f3000 0x16a04f3000(0x16a04f3000) 00000800:00000200:10.0:1547251667.656869:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97baa285e000 0x166285e000(0x166285e000) 00000800:00000200:10.0:1547251667.656869:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b028e16000 0xbe8e16000(0xbe8e16000) 00000800:00000200:10.0:1547251667.656870:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97a601bab000 0x1c1bab000(0x1c1bab000) 00000800:00000200:10.0:1547251667.656871:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97a601b82000 0x1c1b82000(0x1c1b82000) 00000800:00000200:10.0:1547251667.656872:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97baa0f30000 0x1660f30000(0x1660f30000) 00000800:00000200:10.0:1547251667.656873:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ba91a4b000 0x1651a4b000(0x1651a4b000) 00000800:00000200:10.0:1547251667.656874:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba961ae000 0x16561ae000(0x16561ae000) 00000800:00000200:10.0:1547251667.656875:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b80cd1b000 0x13ccd1b000(0x13ccd1b000) 00000800:00000200:10.0:1547251667.656876:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97baa1dbf000 0x1661dbf000(0x1661dbf000) 00000800:00000200:10.0:1547251667.656876:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97badc8bf000 0x169c8bf000(0x169c8bf000) 00000800:00000200:10.0:1547251667.656877:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba9bf33000 0x165bf33000(0x165bf33000) 00000800:00000200:10.0:1547251667.656878:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97baa0e2c000 0x1660e2c000(0x1660e2c000) 00000800:00000200:10.0:1547251667.656879:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97babd0e4000 0x167d0e4000(0x167d0e4000) 00000800:00000200:9.0:1547251667.657450:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657452:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657452:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657453:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657454:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657454:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657455:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657456:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657456:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657457:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657457:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657458:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657459:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657459:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657460:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657460:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657461:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657462:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657462:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657463:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657464:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657464:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657465:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657465:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657466:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657467:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657467:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657468:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657468:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657469:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657470:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657470:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657471:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657471:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:9.0:1547251667.657472:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:9.0:1547251667.657473:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000100:12.0:1547251667.657795:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251667.657798:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251667.657799:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251667.657840:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07fe00] (19)++ 00000800:00000200:13.0:1547251667.657860:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (20)++ 00000800:00000200:12.0:1547251667.657860:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:13.0:1547251667.657866:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (20)-- 00000800:00000200:13.0:1547251667.657868:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (19)-- 00000800:00000200:13.0:1547251667.657869:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (18)-- 00000800:00000200:8.2:1547251667.657996:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07fe00] (17)++ 00000800:00000200:13.0:1547251667.658007:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (18)++ 00000800:00000200:13.0:1547251667.658011:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (19)-- 00000800:00000200:13.0:1547251667.658012:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (18)-- 00000800:00000200:13.0:1547251667.658013:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (17)++ 00000800:00000200:13.0:1547251667.658014:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (18)-- 00000800:00000200:13.0:1547251667.658014:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (17)-- 00000800:00000200:13.0:1547251667.658015:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (16)++ 00000800:00000200:13.0:1547251667.658016:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (17)-- 00000800:00000200:13.0:1547251667.658017:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (16)-- 00000800:00000200:13.0:1547251667.658017:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (15)++ 00000800:00000200:13.0:1547251667.658021:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (16)-- 00000800:00000200:13.0:1547251667.658021:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (15)-- 00000800:00000200:13.0:1547251667.658022:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (14)++ 00000800:00000200:13.0:1547251667.658023:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (15)-- 00000800:00000200:13.0:1547251667.658024:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (14)-- 00000800:00000200:15.0:1547251667.658025:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (13)++ 00000800:00000200:15.0:1547251667.658031:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (14)-- 00000800:00000200:15.0:1547251667.658032:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (13)-- 00000800:00000200:15.0:1547251667.658033:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (12)++ 00000800:00000200:15.0:1547251667.658036:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (13)-- 00000800:00000200:15.0:1547251667.658037:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (12)-- 00000800:00000200:15.0:1547251667.658038:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (11)++ 00000800:00000200:15.0:1547251667.658039:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (12)-- 00000800:00000200:13.0:1547251667.658039:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (12)++ 00000800:00000200:15.0:1547251667.658040:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (12)-- 00000800:00000200:13.0:1547251667.658041:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (11)-- 00000800:00000200:15.0:1547251667.658042:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (11)++ 00000800:00000200:13.0:1547251667.658042:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (11)-- 00000800:00000200:15.0:1547251667.658043:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (10)-- 00000800:00000200:15.0:1547251667.658044:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (10)-- 00000800:00000200:13.0:1547251667.658044:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (10)++ 00000800:00000200:15.0:1547251667.658045:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (9)++ 00000800:00000200:13.0:1547251667.658045:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (9)-- 00000800:00000200:13.0:1547251667.658046:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (9)-- 00000800:00000200:15.0:1547251667.658047:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (8)-- 00000800:00000200:13.0:1547251667.658047:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (8)++ 00000800:00000200:15.0:1547251667.658048:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (8)-- 00000800:00000200:15.0:1547251667.658049:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (7)++ 00000800:00000200:13.0:1547251667.658049:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (7)-- 00000800:00000200:13.0:1547251667.658050:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (7)-- 00000800:00000200:15.0:1547251667.658051:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (6)-- 00000800:00000200:13.0:1547251667.658051:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (6)++ 00000800:00000200:15.0:1547251667.658052:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (6)-- 00000800:00000200:15.0:1547251667.658053:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (5)++ 00000800:00000200:13.0:1547251667.658053:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (5)-- 00000800:00000200:15.0:1547251667.658054:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (6)-- 00000800:00000200:15.0:1547251667.658055:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (4)-- 00000800:00000200:13.0:1547251667.658056:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (3)-- 00000800:00000200:15.0:1547251667.658057:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07fe00] (2)++ 00000800:00000200:15.0:1547251667.658060:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07fe00] (3)-- 00000800:00000200:15.0:1547251667.658060:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (2)-- 00000800:00000200:15.0:1547251667.658062:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07fe00] (1)-- 00000800:00000200:9.0:1547251667.658073:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.658811:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251667.658816:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.658825:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251667.658826:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251667.658832:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251667.658838:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251667.659039:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251667.659047:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b630fa6000 0x11f0fa6000(0x11f0fa6000) 00000800:00000200:10.0:1547251667.659048:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97a65d22a000 0x21d22a000(0x21d22a000) 00000800:00000200:10.0:1547251667.659049:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba92fef000 0x1652fef000(0x1652fef000) 00000800:00000200:10.0:1547251667.659050:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b8247c0000 0x13e47c0000(0x13e47c0000) 00000800:00000200:10.0:1547251667.659050:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b81d4e9000 0x13dd4e9000(0x13dd4e9000) 00000800:00000200:10.0:1547251667.659051:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ae6a83a000 0xa2a83a000(0xa2a83a000) 00000800:00000200:10.0:1547251667.659052:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b82463f000 0x13e463f000(0x13e463f000) 00000800:00000200:10.0:1547251667.659053:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97a65d29d000 0x21d29d000(0x21d29d000) 00000800:00000200:10.0:1547251667.659054:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ad572f9000 0x9172f9000(0x9172f9000) 00000800:00000200:10.0:1547251667.659055:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ae6a970000 0xa2a970000(0xa2a970000) 00000800:00000200:10.0:1547251667.659055:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ba95ba2000 0x1655ba2000(0x1655ba2000) 00000800:00000200:10.0:1547251667.659056:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba99fe5000 0x1659fe5000(0x1659fe5000) 00000800:00000200:10.0:1547251667.659057:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba94159000 0x1654159000(0x1654159000) 00000800:00000200:10.0:1547251667.659058:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97baa13a5000 0x16613a5000(0x16613a5000) 00000800:00000200:10.0:1547251667.659059:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ae5c43e000 0xa1c43e000(0xa1c43e000) 00000800:00000200:10.0:1547251667.659060:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97bae1a95000 0x16a1a95000(0x16a1a95000) 00000800:00000200:10.0:1547251667.659061:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ba912e1000 0x16512e1000(0x16512e1000) 00000800:00000200:10.0:1547251667.659062:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b81d44f000 0x13dd44f000(0x13dd44f000) 00000800:00000200:9.0:1547251667.659611:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659612:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659613:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659613:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659614:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659614:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659615:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659616:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659616:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659617:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659618:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659618:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659619:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659619:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659620:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659621:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659621:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659622:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659622:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659623:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659624:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659624:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659625:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659625:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659626:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659627:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659627:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659628:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659628:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659629:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659630:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659630:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659631:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659632:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251667.659633:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251667.659633:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000100:12.0:1547251667.659970:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251667.659973:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251667.659974:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251667.660012:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:12.0:1547251667.660026:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:13.0:1547251667.660032:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:13.0:1547251667.660038:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:13.0:1547251667.660039:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (19)-- 00000800:00000200:13.0:1547251667.660041:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:8.2:1547251667.660157:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c600] (17)++ 00000800:00000200:13.0:1547251667.660168:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)++ 00000800:00000200:13.0:1547251667.660173:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (19)-- 00000800:00000200:13.0:1547251667.660173:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:13.0:1547251667.660174:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (17)++ 00000800:00000200:13.0:1547251667.660175:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:13.0:1547251667.660176:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (17)-- 00000800:00000200:13.0:1547251667.660177:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (16)++ 00000800:00000200:13.0:1547251667.660177:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (17)-- 00000800:00000200:13.0:1547251667.660178:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (16)-- 00000800:00000200:13.0:1547251667.660179:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)++ 00000800:00000200:13.0:1547251667.660180:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (16)-- 00000800:00000200:13.0:1547251667.660181:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)-- 00000800:00000200:15.0:1547251667.660182:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (14)++ 00000800:00000200:15.0:1547251667.660189:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (15)-- 00000800:00000200:15.0:1547251667.660190:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (14)-- 00000800:00000200:15.0:1547251667.660191:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)++ 00000800:00000200:15.0:1547251667.660194:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (14)-- 00000800:00000200:15.0:1547251667.660195:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:15.0:1547251667.660195:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)++ 00000800:00000200:15.0:1547251667.660197:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:13.0:1547251667.660197:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)++ 00000800:00000200:15.0:1547251667.660198:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:15.0:1547251667.660199:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)++ 00000800:00000200:13.0:1547251667.660199:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (12)-- 00000800:00000200:13.0:1547251667.660201:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)-- 00000800:00000200:15.0:1547251667.660202:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (11)-- 00000800:00000200:13.0:1547251667.660202:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)++ 00000800:00000200:15.0:1547251667.660203:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)-- 00000800:00000200:13.0:1547251667.660204:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (10)-- 00000800:00000200:15.0:1547251667.660205:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (10)++ 00000800:00000200:13.0:1547251667.660205:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (10)-- 00000800:00000200:15.0:1547251667.660206:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (9)-- 00000800:00000200:13.0:1547251667.660207:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (9)++ 00000800:00000200:15.0:1547251667.660208:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (9)-- 00000800:00000200:15.0:1547251667.660209:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)++ 00000800:00000200:13.0:1547251667.660209:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:13.0:1547251667.660210:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:15.0:1547251667.660211:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (7)-- 00000800:00000200:13.0:1547251667.660211:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (7)++ 00000800:00000200:15.0:1547251667.660212:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (7)-- 00000800:00000200:15.0:1547251667.660213:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (6)++ 00000800:00000200:13.0:1547251667.660213:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (6)-- 00000800:00000200:13.0:1547251667.660214:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (6)-- 00000800:00000200:15.0:1547251667.660215:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (5)-- 00000800:00000200:13.0:1547251667.660215:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)++ 00000800:00000200:15.0:1547251667.660216:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)-- 00000800:00000200:13.0:1547251667.660216:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (4)-- 00000800:00000200:15.0:1547251667.660217:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (4)++ 00000800:00000200:13.0:1547251667.660218:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (4)-- 00000800:00000200:15.0:1547251667.660219:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (3)-- 00000800:00000200:13.0:1547251667.660219:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (3)-- 00000800:00000200:15.0:1547251667.660220:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (1)-- 00000800:00000200:9.0:1547251667.660232:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.660974:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251667.660978:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.660986:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251667.660987:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251667.660993:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251667.661000:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251667.661193:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251667.661199:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba9d3ea000 0x165d3ea000(0x165d3ea000) 00000800:00000200:10.0:1547251667.661200:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97baa0e56000 0x1660e56000(0x1660e56000) 00000800:00000200:10.0:1547251667.661201:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97baa0e9e000 0x1660e9e000(0x1660e9e000) 00000800:00000200:10.0:1547251667.661202:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97bae273e000 0x16a273e000(0x16a273e000) 00000800:00000200:10.0:1547251667.661203:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97bae1bf4000 0x16a1bf4000(0x16a1bf4000) 00000800:00000200:10.0:1547251667.661203:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97bade64f000 0x169e64f000(0x169e64f000) 00000800:00000200:10.0:1547251667.661204:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ba927fa000 0x16527fa000(0x16527fa000) 00000800:00000200:10.0:1547251667.661205:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba9a6df000 0x165a6df000(0x165a6df000) 00000800:00000200:10.0:1547251667.661206:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b17b056000 0xd3b056000(0xd3b056000) 00000800:00000200:10.0:1547251667.661207:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b5bdd35000 0x117dd35000(0x117dd35000) 00000800:00000200:10.0:1547251667.661208:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97bae1342000 0x16a1342000(0x16a1342000) 00000800:00000200:10.0:1547251667.661208:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97a8c76a9000 0x4876a9000(0x4876a9000) 00000800:00000200:10.0:1547251667.661209:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ae37822000 0x9f7822000(0x9f7822000) 00000800:00000200:10.0:1547251667.661210:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba9ea1b000 0x165ea1b000(0x165ea1b000) 00000800:00000200:10.0:1547251667.661211:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97babe286000 0x167e286000(0x167e286000) 00000800:00000200:10.0:1547251667.661212:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97baa1673000 0x1661673000(0x1661673000) 00000800:00000200:10.0:1547251667.661213:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97a6c4c9d000 0x284c9d000(0x284c9d000) 00000800:00000200:10.0:1547251667.661214:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97baa1e8b000 0x1661e8b000(0x1661e8b000) 00000800:00000200:9.0:1547251667.661761:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661762:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661763:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661763:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661764:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661764:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661765:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661766:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661766:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661767:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661768:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661768:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661769:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661769:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661770:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661771:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661771:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661772:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661772:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661773:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661774:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661774:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661775:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661775:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661776:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661777:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661777:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661778:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661778:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661779:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661780:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661781:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661781:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661782:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251667.661783:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251667.661783:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000100:12.0:1547251667.662073:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251667.662076:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251667.662077:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251667.662152:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:12.0:1547251667.662157:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:13.0:1547251667.662171:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:13.0:1547251667.662177:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:13.0:1547251667.662178:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (19)-- 00000800:00000200:13.0:1547251667.662180:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (18)-- 00000800:00000200:8.2:1547251667.662289:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c800] (17)++ 00000800:00000200:13.0:1547251667.662301:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (18)++ 00000800:00000200:13.0:1547251667.662305:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (19)-- 00000800:00000200:13.0:1547251667.662306:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (18)-- 00000800:00000200:13.0:1547251667.662307:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (17)++ 00000800:00000200:13.0:1547251667.662308:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (18)-- 00000800:00000200:13.0:1547251667.662308:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (17)-- 00000800:00000200:13.0:1547251667.662309:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (16)++ 00000800:00000200:13.0:1547251667.662310:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (17)-- 00000800:00000200:13.0:1547251667.662311:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (16)-- 00000800:00000200:13.0:1547251667.662312:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (15)++ 00000800:00000200:13.0:1547251667.662315:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (16)-- 00000800:00000200:13.0:1547251667.662315:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (15)-- 00000800:00000200:13.0:1547251667.662316:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (14)++ 00000800:00000200:13.0:1547251667.662318:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (15)-- 00000800:00000200:13.0:1547251667.662318:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (14)-- 00000800:00000200:15.0:1547251667.662319:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (13)++ 00000800:00000200:15.0:1547251667.662326:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (14)-- 00000800:00000200:15.0:1547251667.662327:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (13)-- 00000800:00000200:15.0:1547251667.662328:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (12)++ 00000800:00000200:15.0:1547251667.662331:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (13)-- 00000800:00000200:15.0:1547251667.662332:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (12)-- 00000800:00000200:15.0:1547251667.662333:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (11)++ 00000800:00000200:15.0:1547251667.662334:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (12)-- 00000800:00000200:13.0:1547251667.662334:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (12)++ 00000800:00000200:15.0:1547251667.662335:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (12)-- 00000800:00000200:15.0:1547251667.662336:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (11)++ 00000800:00000200:13.0:1547251667.662336:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (11)-- 00000800:00000200:13.0:1547251667.662337:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (11)-- 00000800:00000200:15.0:1547251667.662338:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (10)-- 00000800:00000200:13.0:1547251667.662338:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (10)++ 00000800:00000200:15.0:1547251667.662339:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (10)-- 00000800:00000200:15.0:1547251667.662340:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (9)++ 00000800:00000200:13.0:1547251667.662340:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (9)-- 00000800:00000200:13.0:1547251667.662341:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (9)-- 00000800:00000200:15.0:1547251667.662342:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (8)-- 00000800:00000200:13.0:1547251667.662342:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (8)++ 00000800:00000200:15.0:1547251667.662343:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (8)-- 00000800:00000200:15.0:1547251667.662344:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (7)++ 00000800:00000200:13.0:1547251667.662344:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (7)-- 00000800:00000200:13.0:1547251667.662345:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (7)-- 00000800:00000200:15.0:1547251667.662346:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (6)-- 00000800:00000200:13.0:1547251667.662346:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (6)++ 00000800:00000200:15.0:1547251667.662347:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (6)-- 00000800:00000200:15.0:1547251667.662348:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (5)++ 00000800:00000200:13.0:1547251667.662348:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (5)-- 00000800:00000200:13.0:1547251667.662349:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (5)-- 00000800:00000200:15.0:1547251667.662350:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (4)-- 00000800:00000200:13.0:1547251667.662350:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (4)++ 00000800:00000200:15.0:1547251667.662351:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (4)-- 00000800:00000200:15.0:1547251667.662352:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (3)-- 00000800:00000200:13.0:1547251667.662352:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (3)-- 00000800:00000200:13.0:1547251667.662353:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (1)-- 00000800:00000200:9.0:1547251667.662364:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.663049:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251667.663052:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.663060:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251667.663062:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251667.663068:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251667.663074:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251667.663307:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251667.663314:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97baa209f000 0x166209f000(0x166209f000) 00000800:00000200:10.0:1547251667.663315:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97a65d2ff000 0x21d2ff000(0x21d2ff000) 00000800:00000200:10.0:1547251667.663316:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97a765da6000 0x325da6000(0x325da6000) 00000800:00000200:10.0:1547251667.663317:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97babd481000 0x167d481000(0x167d481000) 00000800:00000200:10.0:1547251667.663318:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97bade7f9000 0x169e7f9000(0x169e7f9000) 00000800:00000200:10.0:1547251667.663319:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b6268ce000 0x11e68ce000(0x11e68ce000) 00000800:00000200:10.0:1547251667.663320:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97badd67c000 0x169d67c000(0x169d67c000) 00000800:00000200:10.0:1547251667.663321:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba9bae9000 0x165bae9000(0x165bae9000) 00000800:00000200:10.0:1547251667.663322:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b6268c2000 0x11e68c2000(0x11e68c2000) 00000800:00000200:10.0:1547251667.663323:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ad5727d000 0x91727d000(0x91727d000) 00000800:00000200:10.0:1547251667.663323:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b3b5c04000 0xf75c04000(0xf75c04000) 00000800:00000200:10.0:1547251667.663324:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba9ae84000 0x165ae84000(0x165ae84000) 00000800:00000200:10.0:1547251667.663325:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba8b02e000 0x164b02e000(0x164b02e000) 00000800:00000200:10.0:1547251667.663326:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ac55e79000 0x815e79000(0x815e79000) 00000800:00000200:10.0:1547251667.663327:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97babd1e2000 0x167d1e2000(0x167d1e2000) 00000800:00000200:10.0:1547251667.663328:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97a65d0e5000 0x21d0e5000(0x21d0e5000) 00000800:00000200:10.0:1547251667.663328:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97bae1cb6000 0x16a1cb6000(0x16a1cb6000) 00000800:00000200:10.0:1547251667.663329:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97badb957000 0x169b957000(0x169b957000) 00000800:00000200:9.0:1547251667.663862:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663863:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663863:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663864:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663865:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663865:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663866:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663867:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663867:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663868:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663868:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663869:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663870:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663870:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663871:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663871:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663872:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663873:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663873:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663874:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663874:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663875:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663876:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663876:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663877:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663878:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663879:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663879:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663880:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663880:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663881:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663882:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663882:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663883:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251667.663883:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251667.663884:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000100:12.0:1547251667.664215:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251667.664218:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251667.664220:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:12.0:1547251667.664268:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07cc00] (19)-- 00000800:00000200:8.2:1547251667.664271:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07cc00] (18)++ 00000800:00000200:15.0:1547251667.664319:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:15.0:1547251667.664330:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:15.0:1547251667.664331:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (19)-- 00000800:00000200:15.0:1547251667.664332:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (18)-- 00000800:00000200:8.2:1547251667.664416:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07cc00] (17)++ 00000800:00000200:15.0:1547251667.664428:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (18)++ 00000800:00000200:15.0:1547251667.664433:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (19)-- 00000800:00000200:15.0:1547251667.664434:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (18)-- 00000800:00000200:15.0:1547251667.664435:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (17)++ 00000800:00000200:15.0:1547251667.664435:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (18)-- 00000800:00000200:15.0:1547251667.664436:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (17)-- 00000800:00000200:15.0:1547251667.664437:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (16)++ 00000800:00000200:15.0:1547251667.664438:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (17)-- 00000800:00000200:15.0:1547251667.664439:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (16)-- 00000800:00000200:15.0:1547251667.664439:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (15)++ 00000800:00000200:15.0:1547251667.664443:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (16)-- 00000800:00000200:15.0:1547251667.664443:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (15)-- 00000800:00000200:15.0:1547251667.664444:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (14)++ 00000800:00000200:15.0:1547251667.664447:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (15)-- 00000800:00000200:15.0:1547251667.664448:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (14)-- 00000800:00000200:15.0:1547251667.664449:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (13)++ 00000800:00000200:15.0:1547251667.664450:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (14)-- 00000800:00000200:15.0:1547251667.664450:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (13)-- 00000800:00000200:13.0:1547251667.664452:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (12)++ 00000800:00000200:13.0:1547251667.664457:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (13)-- 00000800:00000200:13.0:1547251667.664458:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (12)-- 00000800:00000200:13.0:1547251667.664459:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (11)++ 00000800:00000200:13.0:1547251667.664462:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (12)-- 00000800:00000200:13.0:1547251667.664463:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (11)-- 00000800:00000200:13.0:1547251667.664464:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (10)++ 00000800:00000200:13.0:1547251667.664466:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (11)-- 00000800:00000200:15.0:1547251667.664467:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (11)++ 00000800:00000200:13.0:1547251667.664468:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (11)-- 00000800:00000200:15.0:1547251667.664469:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (10)-- 00000800:00000200:13.0:1547251667.664469:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (10)++ 00000800:00000200:15.0:1547251667.664470:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (10)-- 00000800:00000200:15.0:1547251667.664471:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (9)++ 00000800:00000200:13.0:1547251667.664471:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (9)-- 00000800:00000200:13.0:1547251667.664472:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (9)-- 00000800:00000200:15.0:1547251667.664473:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (8)-- 00000800:00000200:13.0:1547251667.664473:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (8)++ 00000800:00000200:15.0:1547251667.664474:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (8)-- 00000800:00000200:15.0:1547251667.664475:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (7)++ 00000800:00000200:13.0:1547251667.664475:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (7)-- 00000800:00000200:13.0:1547251667.664476:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (7)-- 00000800:00000200:15.0:1547251667.664477:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (6)-- 00000800:00000200:13.0:1547251667.664477:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (6)++ 00000800:00000200:15.0:1547251667.664478:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (6)-- 00000800:00000200:15.0:1547251667.664479:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (5)++ 00000800:00000200:13.0:1547251667.664479:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (5)-- 00000800:00000200:13.0:1547251667.664480:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (5)-- 00000800:00000200:15.0:1547251667.664481:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (4)-- 00000800:00000200:13.0:1547251667.664481:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (4)++ 00000800:00000200:15.0:1547251667.664482:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (4)-- 00000800:00000200:15.0:1547251667.664483:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (3)-- 00000800:00000200:13.0:1547251667.664483:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (3)-- 00000800:00000200:13.0:1547251667.664484:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (1)-- 00000800:00000200:9.0:1547251667.664496:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.665303:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251667.665306:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.665315:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251667.665316:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251667.665322:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251667.665331:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251667.665577:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251667.665598:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97baddebc000 0x169debc000(0x169debc000) 00000800:00000200:10.0:1547251667.665599:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba9ba4b000 0x165ba4b000(0x165ba4b000) 00000800:00000200:10.0:1547251667.665600:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba9bf9c000 0x165bf9c000(0x165bf9c000) 00000800:00000200:10.0:1547251667.665601:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97badd522000 0x169d522000(0x169d522000) 00000800:00000200:10.0:1547251667.665602:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97bad6a8c000 0x1696a8c000(0x1696a8c000) 00000800:00000200:10.0:1547251667.665603:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba9ce49000 0x165ce49000(0x165ce49000) 00000800:00000200:10.0:1547251667.665604:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97bae1cfd000 0x16a1cfd000(0x16a1cfd000) 00000800:00000200:10.0:1547251667.665604:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba90fac000 0x1650fac000(0x1650fac000) 00000800:00000200:10.0:1547251667.665605:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97a601aa3000 0x1c1aa3000(0x1c1aa3000) 00000800:00000200:10.0:1547251667.665606:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba95c64000 0x1655c64000(0x1655c64000) 00000800:00000200:10.0:1547251667.665607:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97bade77b000 0x169e77b000(0x169e77b000) 00000800:00000200:10.0:1547251667.665608:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba957c3000 0x16557c3000(0x16557c3000) 00000800:00000200:10.0:1547251667.665609:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b1231bf000 0xce31bf000(0xce31bf000) 00000800:00000200:10.0:1547251667.665610:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba92af3000 0x1652af3000(0x1652af3000) 00000800:00000200:10.0:1547251667.665611:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97bae04f3000 0x16a04f3000(0x16a04f3000) 00000800:00000200:10.0:1547251667.665612:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97baa285e000 0x166285e000(0x166285e000) 00000800:00000200:10.0:1547251667.665613:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b028e16000 0xbe8e16000(0xbe8e16000) 00000800:00000200:10.0:1547251667.665614:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97a601bab000 0x1c1bab000(0x1c1bab000) 00000800:00000200:9.0:1547251667.666159:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666160:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666161:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666162:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666162:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666163:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666163:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666164:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666165:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666165:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666166:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666166:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666167:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666168:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666168:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666169:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666169:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666170:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666171:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666171:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666172:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666172:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666174:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666174:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666175:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666175:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666176:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666177:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666177:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666178:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666178:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666179:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666179:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666180:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251667.666181:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251667.666181:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000100:12.0:1547251667.666485:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251667.666489:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251667.666491:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251667.666537:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:12.0:1547251667.666561:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:15.0:1547251667.666603:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:15.0:1547251667.666609:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:15.0:1547251667.666610:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (19)-- 00000800:00000200:15.0:1547251667.666612:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (18)-- 00000800:00000200:8.2:1547251667.666678:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07ca00] (17)++ 00000800:00000200:15.0:1547251667.666691:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (18)++ 00000800:00000200:15.0:1547251667.666696:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (19)-- 00000800:00000200:15.0:1547251667.666696:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (18)-- 00000800:00000200:15.0:1547251667.666697:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (17)++ 00000800:00000200:15.0:1547251667.666698:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (18)-- 00000800:00000200:15.0:1547251667.666699:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (17)-- 00000800:00000200:15.0:1547251667.666700:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (16)++ 00000800:00000200:15.0:1547251667.666701:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (17)-- 00000800:00000200:15.0:1547251667.666701:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (16)-- 00000800:00000200:15.0:1547251667.666702:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (15)++ 00000800:00000200:15.0:1547251667.666705:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (16)-- 00000800:00000200:15.0:1547251667.666706:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (15)-- 00000800:00000200:15.0:1547251667.666707:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (14)++ 00000800:00000200:15.0:1547251667.666708:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (15)-- 00000800:00000200:15.0:1547251667.666709:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (14)-- 00000800:00000200:13.0:1547251667.666710:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (13)++ 00000800:00000200:13.0:1547251667.666716:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (14)-- 00000800:00000200:13.0:1547251667.666717:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (13)-- 00000800:00000200:13.0:1547251667.666718:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (12)++ 00000800:00000200:13.0:1547251667.666721:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (13)-- 00000800:00000200:13.0:1547251667.666721:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (12)-- 00000800:00000200:13.0:1547251667.666722:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (11)++ 00000800:00000200:13.0:1547251667.666723:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (12)-- 00000800:00000200:15.0:1547251667.666724:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (12)++ 00000800:00000200:13.0:1547251667.666725:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (12)-- 00000800:00000200:15.0:1547251667.666726:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (11)-- 00000800:00000200:13.0:1547251667.666726:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (11)++ 00000800:00000200:15.0:1547251667.666727:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (11)-- 00000800:00000200:15.0:1547251667.666728:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (10)++ 00000800:00000200:13.0:1547251667.666728:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (10)-- 00000800:00000200:13.0:1547251667.666729:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (10)-- 00000800:00000200:15.0:1547251667.666730:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (9)-- 00000800:00000200:13.0:1547251667.666730:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (9)++ 00000800:00000200:15.0:1547251667.666731:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (9)-- 00000800:00000200:15.0:1547251667.666732:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (8)++ 00000800:00000200:13.0:1547251667.666732:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (8)-- 00000800:00000200:13.0:1547251667.666733:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (8)-- 00000800:00000200:15.0:1547251667.666734:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (7)-- 00000800:00000200:13.0:1547251667.666734:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (7)++ 00000800:00000200:15.0:1547251667.666735:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (7)-- 00000800:00000200:15.0:1547251667.666736:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (6)++ 00000800:00000200:13.0:1547251667.666736:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (6)-- 00000800:00000200:13.0:1547251667.666737:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (6)-- 00000800:00000200:15.0:1547251667.666738:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (5)-- 00000800:00000200:13.0:1547251667.666738:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (5)++ 00000800:00000200:15.0:1547251667.666739:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (5)-- 00000800:00000200:15.0:1547251667.666740:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (4)++ 00000800:00000200:13.0:1547251667.666740:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (4)-- 00000800:00000200:13.0:1547251667.666741:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (4)-- 00000800:00000200:15.0:1547251667.666742:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (3)-- 00000800:00000200:13.0:1547251667.666742:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (3)-- 00000800:00000200:15.0:1547251667.666743:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (1)-- 00000800:00000200:9.0:1547251667.666750:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251667.667614:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000100:00000200:14.0:1547251667.738682:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447376, portal 4 00000100:00000200:14.0:1547251667.738685:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447376, offset 0 00000100:00000200:9.0:1547251667.738686:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447360, portal 4 00000400:00000200:14.0:1547251667.738688:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000100:00000200:9.0:1547251667.738688:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447360, offset 0 00000400:00000200:14.0:1547251667.738691:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251667.738692:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:14.0:1547251667.738693:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23086] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251667.738694:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:14.0:1547251667.738697:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23083] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23086] 00000400:00000200:9.0:1547251667.738697:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23086] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:14.0:1547251667.738699:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251667.738700:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23083] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23086] 00000400:00000200:14.0:1547251667.738701:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251667.738702:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251667.738702:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:14.0:1547251667.738704:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9763, 9764] 00000400:00000200:9.0:1547251667.738704:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251667.738706:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:9.0:1547251667.738706:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251667.738707:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9763, 9764] 00000400:00000200:9.0:1547251667.738709:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:14.0:1547251667.738710:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251667.738713:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:14.0:1547251667.738716:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:9.0:1547251667.738716:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251667.738718:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:14.0:1547251667.738719:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:9.0:1547251667.738719:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:14.0:1547251667.738721:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:9.0:1547251667.738722:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251667.738723:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:9.0:1547251667.738724:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:9.0:1547251667.738727:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:12.2:1547251667.738738:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000100:00000200:3.0:1547251667.738743:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447392, portal 4 00000800:00000200:12.2:1547251667.738747:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000100:00000200:3.0:1547251667.738747:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447392, offset 0 00000400:00000200:3.0:1547251667.738751:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:3.0:1547251667.738753:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:3.0:1547251667.738756:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:254, d:10, s:23088] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251667.738759:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23083] with best_ni 10.0.11.90@o2ib10 [c:254, d:10, s:23088] 00000400:00000200:3.0:1547251667.738761:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:3.0:1547251667.738763:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251667.738764:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251667.738765:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9764, 9765] 00000400:00000200:3.0:1547251667.738767:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:3.0:1547251667.738772:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251667.738775:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:3.0:1547251667.738777:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251667.738779:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:3.0:1547251667.738780:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:3.0:1547251667.738783:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251667.738797:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251667.738801:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251667.738801:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251667.738803:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:0.2:1547251667.738804:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000100:00000200:4.0:1547251667.738805:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a0f00 x1622384012447376/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251667.738807:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf6000 x1622384012447360/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251667.738813:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eabb0 00000400:00000200:4.0:1547251667.738813:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:4.0:1547251667.738814:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251667.738815:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251667.738818:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251667.738818:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251667.738821:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251667.738822:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251667.738823:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:4.0:1547251667.738825:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:5.0:1547251667.738826:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251667.738830:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251667.738831:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:5.0:1547251667.738833:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328def00 x1622384012447392/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251667.738839:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555660 00000400:00000200:5.0:1547251667.738840:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251667.738843:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251667.738845:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:12.2:1547251667.738890:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:0.2:1547251667.738895:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:4.0:1547251667.738903:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251667.738908:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.92@o2ib10 00000800:00000200:5.0:1547251667.738909:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251667.738912:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251667.738913:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251667.738916:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251667.738918:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04e90 00000400:00000200:5.0:1547251667.738920:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04e80 00000400:00000200:4.0:1547251667.738922:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe43f5 [1] + 192 00000400:00000200:5.0:1547251667.738927:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4405 [1] + 192 00000100:00000200:4.0:1547251667.738931:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a0f00 x1622384012447376/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251667.738952:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251667.738954:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf6000 x1622384012447360/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251667.738955:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251667.738957:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:4.0:1547251667.738958:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251667.738960:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251667.738962:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251667.738965:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251667.738967:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000100:00000200:14.0:1547251667.738968:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a0f00 x1622384012447376/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251667.738968:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:5.0:1547251667.738970:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000400:00000200:4.0:1547251667.738970:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04ea0 00000800:00000200:5.0:1547251667.738971:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251667.738973:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:5.0:1547251667.738975:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000100:00000200:14.0:1547251667.738979:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a0f00 x1622384012447376/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251667.738983:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00000100:00000200:9.0:1547251667.738986:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf6000 x1622384012447360/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:14.0:1547251667.738987:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a0f00 x1622384012447376/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251667.738992:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf6000 x1622384012447360/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251667.738996:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea3b8 00000400:00000200:4.0:1547251667.738999:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4415 [1] + 192 00010000:00080000:9.0:1547251667.739000:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf6000 x1622384012447360/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251667.739002:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328def00 x1622384012447392/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251667.739010:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251667.739011:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251667.739012:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251667.739013:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000100:00000200:3.0:1547251667.739019:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328def00 x1622384012447392/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251667.739025:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328def00 x1622384012447392/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251667.739034:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251667.739037:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328def00 x1622384012447392/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251674 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251668.138685:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447408, portal 4 00000100:00000200:9.0:1547251668.138688:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447408, offset 0 00000400:00000200:9.0:1547251668.138690:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251668.138693:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251668.138696:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04eb0 00000400:00000200:9.0:1547251668.138698:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4c49 [1] + 0 00000400:00000200:9.0:1547251668.138700:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ba9d3677f8 00000100:00000200:9.0:1547251668.138702:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251668.138711:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba9d3677f8 00000400:00000200:9.0:1547251668.138712:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251668.138716:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf3900 x1622384012447408/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251675 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251668.138718:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447424, portal 4 00000100:00000200:6.0:1547251668.138721:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447424, offset 0 00000400:00000200:9.0:1547251668.138722:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaee0 00000400:00000200:9.0:1547251668.138723:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251668.138724:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251668.138725:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251668.138727:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251668.138730:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04ec0 00000400:00000200:6.0:1547251668.138733:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2cb5 [1] + 0 00000100:00000200:8.0:1547251668.138734:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447408 00000400:00000200:6.0:1547251668.138735:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ade58 00000100:00000200:6.0:1547251668.138737:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00010000:00000200:8.0:1547251668.138745:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3d8850 x1622384012447408/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:729/0 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:6.0:1547251668.138745:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ade58 00000400:00000200:6.0:1547251668.138746:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251668.138748:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37933c00 x1622384012447424/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251675 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251668.138751:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447408, offset 192 00000400:00000200:8.0:1547251668.138753:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251668.138753:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990c38 00000400:00000200:6.0:1547251668.138754:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251668.138755:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:8.0:1547251668.138756:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251668.138759:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04eb0 00000400:00000200:8.0:1547251668.138761:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf471 [1] + 192 00000100:00000200:8.0:1547251668.138763:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf3900 x1622384012447408/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251675 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251668.138772:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251668.138775:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688908 00000400:00000200:8.0:1547251668.138777:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:5.0:1547251668.138778:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447424 00000400:00000200:8.0:1547251668.138779:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251668.138781:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf3900 x1622384012447408/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251675 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251668.138785:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:9.0:1547251668.138787:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf3900 x1622384012447408/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251675 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00000200:5.0:1547251668.138788:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11ec50 x1622384012447424/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:729/0 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:9.0:1547251668.138792:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea3b8 00000100:00000200:5.0:1547251668.138795:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447424, offset 192 00010000:00080000:9.0:1547251668.138796:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf3900 x1622384012447408/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251675 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251668.138797:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251668.138800:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251668.138803:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04ec0 00000400:00000200:5.0:1547251668.138806:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf481 [1] + 192 00000100:00000200:5.0:1547251668.138809:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37933c00 x1622384012447424/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251675 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251668.138818:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251668.138820:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad550 00000400:00000200:5.0:1547251668.138821:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251668.138822:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251668.138825:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37933c00 x1622384012447424/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251675 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251668.138827:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251668.138829:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37933c00 x1622384012447424/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251675 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251668.138833:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00010000:00080000:6.0:1547251668.138836:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37933c00 x1622384012447424/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251675 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251668.154701:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447440, portal 4 00000100:00000200:6.0:1547251668.154704:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447440, offset 0 00000400:00000200:6.0:1547251668.154708:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251668.154711:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251668.154714:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04ed0 00000400:00000200:6.0:1547251668.154717:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2ce5 [1] + 0 00000400:00000200:6.0:1547251668.154719:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad330 00000100:00000200:6.0:1547251668.154721:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251668.154727:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad330 00000400:00000200:6.0:1547251668.154728:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251668.154730:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37936000 x1622384012447440/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251675 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251668.154735:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990c38 00000400:00000200:6.0:1547251668.154736:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251668.154737:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251668.154765:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447440 00010000:00000200:5.0:1547251668.154774:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11e850 x1622384012447440/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:729/0 lens 224/368 e 0 to 0 dl 1547251674 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251668.154781:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447440, offset 192 00000400:00000200:5.0:1547251668.154783:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251668.154786:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251668.154789:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04ed0 00000400:00000200:5.0:1547251668.154792:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf491 [1] + 192 00000100:00000200:5.0:1547251668.154795:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37936000 x1622384012447440/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251675 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251668.154804:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251668.154807:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad000 00000400:00000200:5.0:1547251668.154807:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251668.154810:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251668.154811:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37936000 x1622384012447440/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251675 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251668.154815:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37936000 x1622384012447440/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251675 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251668.154815:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:6.0:1547251668.154819:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00010000:00080000:6.0:1547251668.154822:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37936000 x1622384012447440/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251675 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:9.0:1547251668.307661:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251668.307674:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251668.307676:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251668.307681:0:12658:0:(o2iblnd_cb.c:3405:kiblnd_check_conns()) conn[ffff97a65d02b200] (20)++ 00000800:00000200:9.0:1547251668.307683:0:12658:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a65d02b200] (21)++ 00000800:00000200:14.0:1547251668.307684:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:9.0:1547251668.307686:0:12658:0:(o2iblnd_cb.c:3440:kiblnd_check_conns()) conn[ffff97a65d02b200] (22)-- 00000800:00000200:14.0:1547251668.307693:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:8.2:1547251668.307719:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d02b200] (21)++ 00000800:00000200:13.0:1547251668.307748:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b200] (22)++ 00000800:00000200:13.0:1547251668.307759:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a65d02b200] (23)-- 00000800:00000200:13.0:1547251668.307761:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b200] (22)-- 00000800:00000200:13.0:1547251668.307763:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b200] (21)-- 00000800:00000200:14.0:1547251668.308215:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:14.0:1547251668.308227:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97bae12b0000 0x16a12b0000(0x16a12b0000) 00000800:00000200:14.0:1547251668.308229:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ad4dfc7000 0x90dfc7000(0x90dfc7000) 00000800:00000200:14.0:1547251668.308230:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b81d511000 0x13dd511000(0x13dd511000) 00000800:00000200:14.0:1547251668.308231:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ae6aa00000 0xa2aa00000(0xa2aa00000) 00000800:00000200:14.0:1547251668.308232:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ae6a81b000 0xa2a81b000(0xa2a81b000) 00000800:00000200:14.0:1547251668.308233:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b81d50e000 0x13dd50e000(0x13dd50e000) 00000800:00000200:14.0:1547251668.308234:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b81d5c6000 0x13dd5c6000(0x13dd5c6000) 00000800:00000200:14.0:1547251668.308235:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ae37bf9000 0x9f7bf9000(0x9f7bf9000) 00000800:00000200:14.0:1547251668.308235:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b6d046c000 0x129046c000(0x129046c000) 00000800:00000200:14.0:1547251668.308236:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ae3793c000 0x9f793c000(0x9f793c000) 00000800:00000200:14.0:1547251668.308237:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97a659df7000 0x219df7000(0x219df7000) 00000800:00000200:14.0:1547251668.308238:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b824692000 0x13e4692000(0x13e4692000) 00000800:00000200:14.0:1547251668.308239:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b8245d7000 0x13e45d7000(0x13e45d7000) 00000800:00000200:14.0:1547251668.308240:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b824738000 0x13e4738000(0x13e4738000) 00000800:00000200:14.0:1547251668.308241:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ae37bee000 0x9f7bee000(0x9f7bee000) 00000800:00000200:14.0:1547251668.308241:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b17b2b9000 0xd3b2b9000(0xd3b2b9000) 00000800:00000200:14.0:1547251668.308242:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ae6a9f5000 0xa2a9f5000(0xa2a9f5000) 00000800:00000200:14.0:1547251668.308243:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ae6a89a000 0xa2a89a000(0xa2a89a000) 00000800:00000200:9.0:1547251668.308950:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308952:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308953:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308954:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308954:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308955:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308955:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308956:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308957:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308957:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308958:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308958:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308959:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308960:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308960:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308961:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308961:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308962:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308963:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308963:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308964:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308964:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308965:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308966:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308966:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308967:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308967:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308968:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308969:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308969:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308970:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308970:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308971:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308972:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:9.0:1547251668.308973:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:9.0:1547251668.308974:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000100:12.0:1547251668.309242:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251668.309245:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251668.309247:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251668.309302:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:12.0:1547251668.309314:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:13.0:1547251668.309327:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (19)++ 00000800:00000200:13.0:1547251668.309335:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (20)-- 00000800:00000200:13.0:1547251668.309336:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (19)-- 00000800:00000200:13.0:1547251668.309338:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (18)-- 00000800:00000200:8.2:1547251668.309385:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ac600] (17)++ 00000800:00000200:13.0:1547251668.309397:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (18)++ 00000800:00000200:13.0:1547251668.309402:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (19)-- 00000800:00000200:13.0:1547251668.309403:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (18)-- 00000800:00000200:13.0:1547251668.309404:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (17)++ 00000800:00000200:13.0:1547251668.309404:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (18)-- 00000800:00000200:13.0:1547251668.309405:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (17)-- 00000800:00000200:13.0:1547251668.309406:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (16)++ 00000800:00000200:13.0:1547251668.309407:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (17)-- 00000800:00000200:13.0:1547251668.309407:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (16)-- 00000800:00000200:13.0:1547251668.309408:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (15)++ 00000800:00000200:13.0:1547251668.309409:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (16)-- 00000800:00000200:13.0:1547251668.309410:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (15)-- 00000800:00000200:15.0:1547251668.309411:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (14)++ 00000800:00000200:15.0:1547251668.309418:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (15)-- 00000800:00000200:15.0:1547251668.309419:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (14)-- 00000800:00000200:15.0:1547251668.309421:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (13)++ 00000800:00000200:15.0:1547251668.309424:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (14)-- 00000800:00000200:15.0:1547251668.309424:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (13)-- 00000800:00000200:15.0:1547251668.309426:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (12)++ 00000800:00000200:15.0:1547251668.309429:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (13)-- 00000800:00000200:15.0:1547251668.309430:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (12)-- 00000800:00000200:15.0:1547251668.309431:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (11)++ 00000800:00000200:15.0:1547251668.309432:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (12)-- 00000800:00000200:15.0:1547251668.309433:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (12)-- 00000800:00000200:13.0:1547251668.309433:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (12)++ 00000800:00000200:13.0:1547251668.309434:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (11)-- 00000800:00000200:15.0:1547251668.309435:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (11)++ 00000800:00000200:15.0:1547251668.309436:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (10)-- 00000800:00000200:13.0:1547251668.309436:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (11)-- 00000800:00000200:15.0:1547251668.309437:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (10)-- 00000800:00000200:13.0:1547251668.309437:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (10)++ 00000800:00000200:13.0:1547251668.309438:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (9)-- 00000800:00000200:15.0:1547251668.309439:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (9)++ 00000800:00000200:15.0:1547251668.309440:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (8)-- 00000800:00000200:13.0:1547251668.309440:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (9)-- 00000800:00000200:15.0:1547251668.309441:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (8)-- 00000800:00000200:13.0:1547251668.309441:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (8)++ 00000800:00000200:13.0:1547251668.309442:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (7)-- 00000800:00000200:15.0:1547251668.309443:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (7)++ 00000800:00000200:13.0:1547251668.309443:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (7)-- 00000800:00000200:15.0:1547251668.309444:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (6)-- 00000800:00000200:15.0:1547251668.309445:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (6)-- 00000800:00000200:13.0:1547251668.309445:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (6)++ 00000800:00000200:13.0:1547251668.309446:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (5)-- 00000800:00000200:15.0:1547251668.309447:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (5)++ 00000800:00000200:13.0:1547251668.309447:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (5)-- 00000800:00000200:15.0:1547251668.309448:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (4)-- 00000800:00000200:15.0:1547251668.309449:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (4)-- 00000800:00000200:13.0:1547251668.309449:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ac600] (4)++ 00000800:00000200:13.0:1547251668.309450:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ac600] (3)-- 00000800:00000200:15.0:1547251668.309451:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (3)-- 00000800:00000200:13.0:1547251668.309452:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ac600] (1)-- 00000800:00000200:9.0:1547251668.309460:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251668.310062:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251669.307659:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251669.307673:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251669.307675:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:14.0:1547251669.307682:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:14.0:1547251669.307691:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:14.0:1547251669.308253:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:14.0:1547251669.308265:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97a49cb7c000 0x5cb7c000(0x5cb7c000) 00000800:00000200:14.0:1547251669.308267:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97a49cb7d000 0x5cb7d000(0x5cb7d000) 00000800:00000200:14.0:1547251669.308269:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97a49cb7e000 0x5cb7e000(0x5cb7e000) 00000800:00000200:14.0:1547251669.308270:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97a49cb7f000 0x5cb7f000(0x5cb7f000) 00000800:00000200:14.0:1547251669.308270:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97a481208000 0x41208000(0x41208000) 00000800:00000200:14.0:1547251669.308271:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97a481209000 0x41209000(0x41209000) 00000800:00000200:14.0:1547251669.308272:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97a48120a000 0x4120a000(0x4120a000) 00000800:00000200:14.0:1547251669.308273:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97a48120b000 0x4120b000(0x4120b000) 00000800:00000200:14.0:1547251669.308274:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97a4af056000 0x6f056000(0x6f056000) 00000800:00000200:14.0:1547251669.308275:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97a4841e3000 0x441e3000(0x441e3000) 00000800:00000200:14.0:1547251669.308275:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97a4bb8fa000 0x7b8fa000(0x7b8fa000) 00000800:00000200:14.0:1547251669.308276:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97a4bb979000 0x7b979000(0x7b979000) 00000800:00000200:14.0:1547251669.308277:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97a48128a000 0x4128a000(0x4128a000) 00000800:00000200:14.0:1547251669.308278:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97a474626000 0x34626000(0x34626000) 00000800:00000200:14.0:1547251669.308279:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97a47e82c000 0x3e82c000(0x3e82c000) 00000800:00000200:14.0:1547251669.308280:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97a48121d000 0x4121d000(0x4121d000) 00000800:00000200:14.0:1547251669.308281:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97a48121e000 0x4121e000(0x4121e000) 00000800:00000200:14.0:1547251669.308281:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97a4b2911000 0x72911000(0x72911000) 00000800:00000200:9.0:1547251669.309019:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309021:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309022:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309023:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309023:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309024:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309024:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309025:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309026:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309026:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309027:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309027:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309028:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309029:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309029:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309030:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309030:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309031:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309032:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309032:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309033:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309033:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309034:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309035:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309035:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309036:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309036:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309037:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309038:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309038:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309039:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309040:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309041:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309041:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:9.0:1547251669.309042:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:9.0:1547251669.309043:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000100:12.0:1547251669.309318:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251669.309321:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251669.309323:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251669.309368:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:12.0:1547251669.309384:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:15.0:1547251669.309425:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (19)++ 00000800:00000200:15.0:1547251669.309434:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (20)-- 00000800:00000200:15.0:1547251669.309435:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (19)-- 00000800:00000200:15.0:1547251669.309436:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (18)-- 00000800:00000200:8.2:1547251669.309502:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ade00] (17)++ 00000800:00000200:15.0:1547251669.309514:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (18)++ 00000800:00000200:15.0:1547251669.309518:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (19)-- 00000800:00000200:15.0:1547251669.309519:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (18)-- 00000800:00000200:15.0:1547251669.309520:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (17)++ 00000800:00000200:15.0:1547251669.309521:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (18)-- 00000800:00000200:15.0:1547251669.309521:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (17)-- 00000800:00000200:15.0:1547251669.309522:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (16)++ 00000800:00000200:15.0:1547251669.309523:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (17)-- 00000800:00000200:15.0:1547251669.309523:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (16)-- 00000800:00000200:15.0:1547251669.309524:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (15)++ 00000800:00000200:15.0:1547251669.309526:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (16)-- 00000800:00000200:15.0:1547251669.309526:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (15)-- 00000800:00000200:13.0:1547251669.309527:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (14)++ 00000800:00000200:13.0:1547251669.309534:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (15)-- 00000800:00000200:13.0:1547251669.309534:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (14)-- 00000800:00000200:13.0:1547251669.309537:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (13)++ 00000800:00000200:13.0:1547251669.309540:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (14)-- 00000800:00000200:13.0:1547251669.309541:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (13)-- 00000800:00000200:13.0:1547251669.309541:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (12)++ 00000800:00000200:15.0:1547251669.309543:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (13)++ 00000800:00000200:13.0:1547251669.309543:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (13)-- 00000800:00000200:13.0:1547251669.309544:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (13)-- 00000800:00000200:15.0:1547251669.309545:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (12)-- 00000800:00000200:13.0:1547251669.309545:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (12)++ 00000800:00000200:13.0:1547251669.309546:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (13)-- 00000800:00000200:15.0:1547251669.309547:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (10)-- 00000800:00000200:13.0:1547251669.309547:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (12)-- 00000800:00000200:13.0:1547251669.309548:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (9)++ 00000800:00000200:13.0:1547251669.309551:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (10)-- 00000800:00000200:13.0:1547251669.309552:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (9)-- 00000800:00000200:13.0:1547251669.309553:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (8)++ 00000800:00000200:15.0:1547251669.309554:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (9)++ 00000800:00000200:13.0:1547251669.309554:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (9)-- 00000800:00000200:13.0:1547251669.309555:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (9)-- 00000800:00000200:15.0:1547251669.309556:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (8)-- 00000800:00000200:13.0:1547251669.309556:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (8)++ 00000800:00000200:15.0:1547251669.309557:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (8)-- 00000800:00000200:15.0:1547251669.309558:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (7)++ 00000800:00000200:13.0:1547251669.309558:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (7)-- 00000800:00000200:13.0:1547251669.309559:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (7)-- 00000800:00000200:15.0:1547251669.309560:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (6)-- 00000800:00000200:15.0:1547251669.309561:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (6)-- 00000800:00000200:13.0:1547251669.309561:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (6)++ 00000800:00000200:13.0:1547251669.309562:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (5)-- 00000800:00000200:15.0:1547251669.309563:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (5)++ 00000800:00000200:13.0:1547251669.309563:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (5)-- 00000800:00000200:15.0:1547251669.309564:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (4)-- 00000800:00000200:15.0:1547251669.309565:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (4)-- 00000800:00000200:13.0:1547251669.309565:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ade00] (4)++ 00000800:00000200:13.0:1547251669.309566:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ade00] (3)-- 00000800:00000200:15.0:1547251669.309567:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (3)-- 00000800:00000200:13.0:1547251669.309568:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ade00] (1)-- 00000800:00000200:9.0:1547251669.309580:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251669.310342:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000100:00000200:11.0:1547251670.170681:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447456, portal 4 00000100:00000200:11.0:1547251670.170684:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447456, offset 0 00000400:00000200:11.0:1547251670.170687:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:11.0:1547251670.170690:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:11.0:1547251670.170692:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23088] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:11.0:1547251670.170695:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23084] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23088] 00000400:00000200:11.0:1547251670.170697:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:11.0:1547251670.170699:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251670.170700:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251670.170701:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9765, 9766] 00000400:00000200:11.0:1547251670.170703:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:11.0:1547251670.170708:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251670.170710:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000100:00000200:7.0:1547251670.170711:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447472, portal 4 00000800:00000200:11.0:1547251670.170713:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000100:00000200:7.0:1547251670.170714:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447472, offset 0 00000800:00000200:11.0:1547251670.170715:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:11.0:1547251670.170716:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:7.0:1547251670.170717:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:11.0:1547251670.170718:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:7.0:1547251670.170720:0:12674:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000100:00000200:2.0:1547251670.170721:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447488, portal 4 00000400:00000200:7.0:1547251670.170723:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23089] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:2.0:1547251670.170725:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447488, offset 0 00000800:00000200:7.2:1547251670.170726:0:12674:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:2.0:1547251670.170729:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:2.0:1547251670.170732:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:2.0:1547251670.170734:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23089] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251670.170738:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23084] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23089] 00000400:00000200:2.0:1547251670.170740:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251670.170741:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251670.170743:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251670.170744:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23085] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23089] 00000400:00000200:2.0:1547251670.170744:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9766, 9766] 00000400:00000200:7.0:1547251670.170746:0:12674:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251670.170747:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:7.0:1547251670.170748:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251670.170749:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251670.170751:0:12674:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9766, 9767] 00000400:00000200:2.0:1547251670.170752:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000400:00000200:7.0:1547251670.170753:0:12674:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000800:00000200:2.0:1547251670.170755:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:7.0:1547251670.170758:0:12674:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251670.170758:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251670.170759:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.0:1547251670.170761:0:12674:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:2.0:1547251670.170761:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:2.0:1547251670.170763:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:7.0:1547251670.170764:0:12674:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:7.0:1547251670.170765:0:12674:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:7.0:1547251670.170766:0:12674:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:7.0:1547251670.170769:0:12674:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:9.2:1547251670.170779:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:9.2:1547251670.170784:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:15.0:1547251670.170785:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:15.0:1547251670.170789:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:15.0:1547251670.170792:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07d700 x1622384012447456/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251677 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251670.170797:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000400:00000200:15.0:1547251670.170798:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bcc0 00000400:00000200:15.0:1547251670.170799:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:13.0:1547251670.170800:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000400:00000200:15.0:1547251670.170801:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000100:00000200:13.0:1547251670.170803:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b6600 x1622384012447488/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251677 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251670.170804:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:15.0:1547251670.170806:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:15.0:1547251670.170808:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000400:00000200:13.0:1547251670.170809:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06aaa0 00000100:00000200:15.0:1547251670.170810:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df0600 x1622384012447472/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251677 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251670.170810:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251670.170812:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:15.0:1547251670.170814:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea770 00000400:00000200:15.0:1547251670.170815:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:13.0:1547251670.170815:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:15.0:1547251670.170817:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251670.170817:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:13.0:1547251670.170818:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:15.0:1547251670.170819:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:13.0:1547251670.170820:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:9.2:1547251670.170907:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:7.2:1547251670.170914:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:15.0:1547251670.170920:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:15.0:1547251670.170923:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.93@o2ib10 00000800:00000200:9.2:1547251670.170923:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000400:00000200:15.0:1547251670.170926:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:13.0:1547251670.170927:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:15.0:1547251670.170930:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04ee0 00000800:00000200:13.0:1547251670.170931:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251670.170933:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7ff8d [1] + 192 00000400:00000200:13.0:1547251670.170934:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000100:00000200:15.0:1547251670.170937:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07d700 x1622384012447456/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251677 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251670.170938:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04ef0 00000400:00000200:15.0:1547251670.170960:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251670.170962:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000400:00000200:13.0:1547251670.170962:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7ff9d [1] + 192 00000800:00000200:15.0:1547251670.170964:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:15.0:1547251670.170965:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:13.0:1547251670.170966:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df0600 x1622384012447472/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251677 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251670.170967:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:15.0:1547251670.170968:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251670.170970:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000400:00000200:15.0:1547251670.170973:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251670.170975:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04f00 00000400:00000200:13.0:1547251670.170976:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251670.170979:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000100:00000200:11.0:1547251670.170979:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07d700 x1622384012447456/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251677 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251670.170980:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:13.0:1547251670.170981:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:13.0:1547251670.170982:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:13.0:1547251670.170984:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:11.0:1547251670.170984:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07d700 x1622384012447456/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251677 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251670.170988:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b908 00000400:00000200:15.0:1547251670.170991:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7ffad [1] + 192 00010000:00080000:11.0:1547251670.170992:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b17b07d700 x1622384012447456/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251677 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251670.170994:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df0600 x1622384012447472/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251677 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251670.171000:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df0600 x1622384012447472/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251677 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251670.171004:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea330 00000100:00000200:15.0:1547251670.171006:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b6600 x1622384012447488/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251677 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:7.0:1547251670.171008:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a765df0600 x1622384012447472/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251677 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:15.0:1547251670.171014:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251670.171015:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251670.171016:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:15.0:1547251670.171017:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:2.0:1547251670.171023:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b6600 x1622384012447488/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251677 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251670.171028:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b6600 x1622384012447488/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251677 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251670.171033:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a5d8 00010000:00080000:2.0:1547251670.171036:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b6600 x1622384012447488/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251677 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:9.0:1547251670.307660:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251670.307674:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251670.307675:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000400:9.0:1547251670.307682:0:12658:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.2.10.184@o2ib10: 0 seconds 00000800:00000200:14.0:1547251670.307683:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000400:00000200:9.0:1547251670.307688:0:12658:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = -110, hstatus = 6 00000400:00000200:9.0:1547251670.307691:0:12658:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.2.10.184@o2ib10: PUT: LOCAL_TIMEOUT 00000800:00000200:14.0:1547251670.307693:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000400:00000100:9.0:1547251670.307693:0:12658:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.2.10.184@o2ib10 exceeded retry count 0 00000400:00000200:9.0:1547251670.307696:0:12658:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06acc0 00000800:00000200:14.0:1547251670.308224:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:14.0:1547251670.308235:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ae6ab11000 0xa2ab11000(0xa2ab11000) 00000800:00000200:14.0:1547251670.308237:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b81d40b000 0x13dd40b000(0x13dd40b000) 00000800:00000200:14.0:1547251670.308238:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b81d689000 0x13dd689000(0x13dd689000) 00000800:00000200:14.0:1547251670.308239:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b81d638000 0x13dd638000(0x13dd638000) 00000800:00000200:14.0:1547251670.308239:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b9072d3000 0x14c72d3000(0x14c72d3000) 00000800:00000200:14.0:1547251670.308240:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ae6ab74000 0xa2ab74000(0xa2ab74000) 00000800:00000200:14.0:1547251670.308241:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ae6a9aa000 0xa2a9aa000(0xa2a9aa000) 00000800:00000200:14.0:1547251670.308242:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b81d767000 0x13dd767000(0x13dd767000) 00000800:00000200:14.0:1547251670.308243:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ae6a97a000 0xa2a97a000(0xa2a97a000) 00000800:00000200:14.0:1547251670.308244:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97a65d357000 0x21d357000(0x21d357000) 00000800:00000200:14.0:1547251670.308245:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b17b3df000 0xd3b3df000(0xd3b3df000) 00000800:00000200:14.0:1547251670.308246:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b17b107000 0xd3b107000(0xd3b107000) 00000800:00000200:14.0:1547251670.308247:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b17b3a2000 0xd3b3a2000(0xd3b3a2000) 00000800:00000200:14.0:1547251670.308248:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b8247cb000 0x13e47cb000(0x13e47cb000) 00000800:00000200:14.0:1547251670.308249:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ae6aa9f000 0xa2aa9f000(0xa2aa9f000) 00000800:00000200:14.0:1547251670.308250:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ae6aaa0000 0xa2aaa0000(0xa2aaa0000) 00000800:00000200:14.0:1547251670.308251:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b17b185000 0xd3b185000(0xd3b185000) 00000800:00000200:14.0:1547251670.308252:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b4f05eb000 0x10b05eb000(0x10b05eb000) 00000800:00000200:9.0:1547251670.308936:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308938:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308939:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308940:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308940:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308941:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308941:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308942:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308950:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308951:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308951:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308952:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308952:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308953:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308954:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308954:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308955:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308955:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308956:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308957:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308958:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308959:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308959:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308960:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308960:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308961:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308962:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308962:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308963:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308963:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308964:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308965:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308965:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308966:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:9.0:1547251670.308967:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:9.0:1547251670.308967:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000100:12.0:1547251670.309238:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: don't reconnect (no need), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251670.309241:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251670.309243:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:12.0:1547251670.309245:0:70:0:(o2iblnd.c:417:kiblnd_unlink_peer_locked()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (2)-- 00000400:00000200:12.0:1547251670.309249:0:70:0:(router.c:1756:lnet_notify()) 10.1.11.90@o2ib10 notifying 10.2.10.184@o2ib10: down 00000400:00000200:12.0:1547251670.309250:0:70:0:(router.c:120:lnet_notify_locked()) Old news 00000800:00000200:8.2:1547251670.309284:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:12.0:1547251670.309302:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:13.0:1547251670.309309:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (19)++ 00000800:00000200:13.0:1547251670.309317:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (20)-- 00000800:00000200:13.0:1547251670.309319:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (19)-- 00000800:00000200:13.0:1547251670.309321:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (18)-- 00000800:00000200:8.2:1547251670.309429:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ae800] (17)++ 00000800:00000200:13.0:1547251670.309442:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (18)++ 00000800:00000200:13.0:1547251670.309446:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (19)-- 00000800:00000200:13.0:1547251670.309447:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (18)-- 00000800:00000200:13.0:1547251670.309448:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (17)++ 00000800:00000200:13.0:1547251670.309449:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (18)-- 00000800:00000200:13.0:1547251670.309449:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (17)-- 00000800:00000200:13.0:1547251670.309450:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (16)++ 00000800:00000200:13.0:1547251670.309451:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (17)-- 00000800:00000200:13.0:1547251670.309451:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (16)-- 00000800:00000200:13.0:1547251670.309452:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (15)++ 00000800:00000200:13.0:1547251670.309454:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (16)-- 00000800:00000200:13.0:1547251670.309454:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (15)-- 00000800:00000200:15.0:1547251670.309455:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (14)++ 00000800:00000200:15.0:1547251670.309463:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (15)-- 00000800:00000200:15.0:1547251670.309464:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (14)-- 00000800:00000200:15.0:1547251670.309465:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (13)++ 00000800:00000200:15.0:1547251670.309468:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (14)-- 00000800:00000200:15.0:1547251670.309469:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (13)-- 00000800:00000200:15.0:1547251670.309469:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (12)++ 00000800:00000200:15.0:1547251670.309471:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (13)-- 00000800:00000200:13.0:1547251670.309471:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (13)++ 00000800:00000200:15.0:1547251670.309472:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (13)-- 00000800:00000200:15.0:1547251670.309473:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (12)++ 00000800:00000200:13.0:1547251670.309473:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (12)-- 00000800:00000200:13.0:1547251670.309474:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (12)-- 00000800:00000200:15.0:1547251670.309475:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (11)-- 00000800:00000200:13.0:1547251670.309475:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (11)++ 00000800:00000200:15.0:1547251670.309476:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (11)-- 00000800:00000200:15.0:1547251670.309477:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (10)++ 00000800:00000200:13.0:1547251670.309477:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (10)-- 00000800:00000200:13.0:1547251670.309478:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (10)-- 00000800:00000200:15.0:1547251670.309479:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (9)-- 00000800:00000200:13.0:1547251670.309479:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (9)++ 00000800:00000200:15.0:1547251670.309481:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (9)-- 00000800:00000200:15.0:1547251670.309482:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (8)++ 00000800:00000200:13.0:1547251670.309482:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (8)-- 00000800:00000200:13.0:1547251670.309483:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (8)-- 00000800:00000200:15.0:1547251670.309484:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (7)-- 00000800:00000200:13.0:1547251670.309484:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (7)++ 00000800:00000200:13.0:1547251670.309485:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (8)-- 00000800:00000200:13.0:1547251670.309486:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (7)-- 00000800:00000200:15.0:1547251670.309487:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (5)-- 00000800:00000200:13.0:1547251670.309487:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (4)++ 00000800:00000200:13.0:1547251670.309491:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (5)-- 00000800:00000200:13.0:1547251670.309492:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (4)-- 00000800:00000200:13.0:1547251670.309492:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (3)++ 00000800:00000200:13.0:1547251670.309493:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (4)-- 00000800:00000200:13.0:1547251670.309494:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (3)-- 00000800:00000200:15.0:1547251670.309495:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ae800] (2)++ 00000800:00000200:15.0:1547251670.309498:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ae800] (3)-- 00000800:00000200:15.0:1547251670.309499:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (2)-- 00000800:00000200:15.0:1547251670.309500:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ae800] (1)-- 00000800:00000200:9.0:1547251670.310408:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0880] -> 10.2.10.184@o2ib10 (1)-- 00000100:00000200:8.0:1547251672.282682:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447504, portal 4 00000100:00000200:8.0:1547251672.282685:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447504, offset 0 00000400:00000200:8.0:1547251672.282688:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251672.282691:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251672.282693:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23089] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251672.282696:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23086] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23089] 00000400:00000200:8.0:1547251672.282698:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:8.0:1547251672.282700:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251672.282701:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251672.282702:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9772, 9771] 00000400:00000200:8.0:1547251672.282704:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:8.0:1547251672.282708:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000100:00000200:15.0:1547251672.282711:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447520, portal 4 00000800:00000200:8.0:1547251672.282711:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000100:00000200:15.0:1547251672.282714:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447520, offset 0 00000800:00000200:8.0:1547251672.282714:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251672.282715:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:8.0:1547251672.282716:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:15.0:1547251672.282717:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:8.0:1547251672.282719:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:15.0:1547251672.282720:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:15.0:1547251672.282723:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23089] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251672.282726:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23087] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23089] 00000400:00000200:15.0:1547251672.282728:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:15.0:1547251672.282730:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251672.282731:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251672.282732:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9772, 9772] 00000400:00000200:15.0:1547251672.282734:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:15.0:1547251672.282739:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251672.282741:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:15.0:1547251672.282744:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251672.282746:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:15.0:1547251672.282747:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000800:00000200:15.0:1547251672.282749:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:7.2:1547251672.282768:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000100:00000200:2.0:1547251672.282783:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447536, portal 4 00000800:00000200:11.0:1547251672.282785:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000100:00000200:2.0:1547251672.282786:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447536, offset 0 00000400:00000200:2.0:1547251672.282789:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:2.0:1547251672.282792:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:11.0:1547251672.282794:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000400:00000200:2.0:1547251672.282794:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23090] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251672.282797:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23087] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23090] 00000100:00000200:11.0:1547251672.282799:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824755100 x1622384012447504/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251672.282800:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251672.282801:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251672.282803:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251672.282804:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9773, 9772] 00000400:00000200:2.0:1547251672.282806:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251672.282807:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688bb0 00000400:00000200:11.0:1547251672.282809:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:2.0:1547251672.282811:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000400:00000200:11.0:1547251672.282812:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251672.282812:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:2.0:1547251672.282813:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:2.0:1547251672.282816:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251672.282817:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:2.0:1547251672.282818:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:5.2:1547251672.282819:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00000800:00000200:2.0:1547251672.282819:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:2.0:1547251672.282823:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:7.2:1547251672.282835:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251672.282841:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000800:00000200:10.0:1547251672.282843:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251672.282848:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000100:00000200:10.0:1547251672.282848:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cab300 x1622384012447520/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251672.282849:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000100:00000200:11.0:1547251672.282853:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f4200 x1622384012447536/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251672.282854:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1990 00000400:00000200:10.0:1547251672.282856:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251672.282859:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06aa18 00000400:00000200:10.0:1547251672.282859:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251672.282860:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251672.282861:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:11.0:1547251672.282863:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251672.282863:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:10.0:1547251672.282864:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251672.282865:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:7.2:1547251672.282891:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:5.2:1547251672.282896:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251672.282902:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:10.0:1547251672.282904:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251672.282908:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000800:00000200:11.0:1547251672.282909:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251672.282912:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251672.282913:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251672.282916:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251672.282917:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04f20 00000400:00000200:11.0:1547251672.282920:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04f10 00000400:00000200:10.0:1547251672.282921:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7569 [1] + 192 00000400:00000200:11.0:1547251672.282925:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7549 [1] + 192 00000100:00000200:10.0:1547251672.282929:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cab300 x1622384012447520/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251672.282941:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251672.282942:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824755100 x1622384012447504/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251672.282952:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251672.282954:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251672.282955:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:10.0:1547251672.282957:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251672.282959:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251672.282962:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:10.0:1547251672.282963:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000100:00000200:15.0:1547251672.282964:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cab300 x1622384012447520/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251672.282964:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251672.282966:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251672.282967:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251672.282967:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04f30 00000800:00000200:11.0:1547251672.282969:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:15.0:1547251672.282970:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cab300 x1622384012447520/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251672.282970:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:11.0:1547251672.282972:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:15.0:1547251672.282974:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb15d8 00010000:00080000:15.0:1547251672.282978:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cab300 x1622384012447520/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251672.282978:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7579 [1] + 192 00000100:00000200:8.0:1547251672.282983:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824755100 x1622384012447504/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251672.282988:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824755100 x1622384012447504/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251672.282992:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6882a8 00000100:00000200:10.0:1547251672.282995:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f4200 x1622384012447536/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251672.282996:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824755100 x1622384012447504/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:10.0:1547251672.283006:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251672.283008:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251672.283009:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251672.283011:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:2.0:1547251672.283015:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f4200 x1622384012447536/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251672.283021:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f4200 x1622384012447536/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251672.283025:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a5d8 00010000:00080000:2.0:1547251672.283029:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f4200 x1622384012447536/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251672.746682:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447552, portal 4 00000100:00000200:15.0:1547251672.746684:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447568, portal 4 00000100:00000200:8.0:1547251672.746685:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447552, offset 0 00000100:00000200:15.0:1547251672.746687:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447568, offset 0 00000400:00000200:8.0:1547251672.746688:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:15.0:1547251672.746690:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251672.746692:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:15.0:1547251672.746694:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251672.746694:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23090] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:2.0:1547251672.746694:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447584, portal 4 00000400:00000200:15.0:1547251672.746696:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23090] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251672.746697:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23088] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23090] 00000100:00000200:2.0:1547251672.746697:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447584, offset 0 00000400:00000200:15.0:1547251672.746699:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23088] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23090] 00000400:00000200:8.0:1547251672.746700:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251672.746700:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251672.746701:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251672.746702:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:15.0:1547251672.746703:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251672.746703:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251672.746703:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251672.746704:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9765, 9765] 00000400:00000200:15.0:1547251672.746705:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251672.746706:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23090] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251672.746707:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9765, 9765] 00000400:00000200:8.0:1547251672.746707:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:15.0:1547251672.746709:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:2.0:1547251672.746709:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23090] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23090] 00000400:00000200:8.0:1547251672.746712:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:2.0:1547251672.746712:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:2.0:1547251672.746713:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251672.746714:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:2.0:1547251672.746715:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251672.746716:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9766, 9766] 00000800:00000200:8.0:1547251672.746717:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:15.0:1547251672.746718:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:2.0:1547251672.746718:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:8.0:1547251672.746719:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:8.0:1547251672.746720:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:15.0:1547251672.746721:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:8.0:1547251672.746722:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:15.0:1547251672.746724:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:2.0:1547251672.746724:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251672.746726:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:15.0:1547251672.746727:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:2.0:1547251672.746727:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:2.0:1547251672.746729:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251672.746730:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:2.0:1547251672.746731:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:2.0:1547251672.746732:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:0.2:1547251672.746733:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:2.0:1547251672.746735:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:0.2:1547251672.746741:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:12.2:1547251672.746751:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251672.746790:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:5.0:1547251672.746794:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:4.0:1547251672.746794:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251672.746797:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:5.0:1547251672.746799:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824750000 x1622384012447552/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251672.746800:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cace00 x1622384012447568/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251672.746806:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688bb0 00000400:00000200:4.0:1547251672.746806:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1dd0 00000400:00000200:5.0:1547251672.746808:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251672.746808:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251672.746811:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251672.746811:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251672.746814:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251672.746816:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251672.746817:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251672.746817:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251672.746818:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251672.746819:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:5.0:1547251672.746820:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:4.0:1547251672.746821:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f3f00 x1622384012447584/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251672.746827:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a4c8 00000400:00000200:4.0:1547251672.746828:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251672.746831:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251672.746833:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:12.2:1547251672.746903:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:0.2:1547251672.746909:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:5.0:1547251672.746918:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251672.746921:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.92@o2ib10 00000800:00000200:0.2:1547251672.746922:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:4.0:1547251672.746923:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000400:00000200:5.0:1547251672.746926:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251672.746926:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251672.746930:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251672.746931:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04f40 00000400:00000200:4.0:1547251672.746934:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04f60 00000400:00000200:5.0:1547251672.746936:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4495 [1] + 192 00000400:00000200:4.0:1547251672.746941:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe44b5 [1] + 192 00000100:00000200:5.0:1547251672.746953:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824750000 x1622384012447552/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251672.746966:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251672.746968:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000100:00000200:4.0:1547251672.746968:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f3f00 x1622384012447584/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251672.746970:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251672.746972:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251672.746974:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:5.0:1547251672.746975:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251672.746978:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251672.746979:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251672.746981:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824750000 x1622384012447552/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251672.746981:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000400:00000200:5.0:1547251672.746982:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251672.746984:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:4.0:1547251672.746985:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:5.0:1547251672.746986:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04f50 00000100:00000200:8.0:1547251672.746987:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824750000 x1622384012447552/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251672.746987:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251672.746988:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:8.0:1547251672.746991:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6882a8 00000100:00000200:2.0:1547251672.746994:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f3f00 x1622384012447584/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251672.746995:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824750000 x1622384012447552/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:2.0:1547251672.747000:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f3f00 x1622384012447584/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251672.747004:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a5d8 00000400:00000200:5.0:1547251672.747007:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe44a5 [1] + 192 00010000:00080000:2.0:1547251672.747008:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f3f00 x1622384012447584/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251672.747011:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cace00 x1622384012447568/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251672.747019:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251672.747021:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251672.747022:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251672.747023:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:15.0:1547251672.747066:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cace00 x1622384012447568/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:15.0:1547251672.747072:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cace00 x1622384012447568/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251672.747077:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb15d8 00010000:00080000:15.0:1547251672.747081:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cace00 x1622384012447568/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251679 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:0.2:1547251672.780583:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b81d647a00] (20)++ 00000800:00000200:13.0:1547251672.780637:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:13.0:1547251672.780647:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.10.184@o2ib10 00000400:00000200:13.0:1547251672.780651:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.10.184@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251672.780655:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.10.184@o2ib10 of length 328 into portal 26 MB=0x5c3934d100050 00000400:00000200:13.0:1547251672.780658:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.10.184@o2ib10 of length 328/328 into md 0xd5599 [1] + 0 00000400:00000200:13.0:1547251672.780661:0:12665:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52df68 00000100:00000200:13.0:1547251672.780663:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:13.0:1547251672.780668:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52df68 00000400:00000200:13.0:1547251672.780669:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251672.780671:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (22)++ 00000800:00000200:13.0:1547251672.780673:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (23)-- 00000800:00000200:13.0:1547251672.780673:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)-- 00000800:00000200:13.0:1547251672.780675:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (21)-- 00000100:00000200:2.0:1547251672.780729:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622412304056400 00010000:00000200:2.0:1547251672.780742:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692feb050 x1622412304056400/t0(0) o101->8534aa65-7ac8-198c-0718-3e6f83d57705@10.0.10.184@o2ib10:733/0 lens 328/344 e 0 to 0 dl 1547251678 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:2.0:1547251672.780748:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 312 bytes to portal 25, xid 1622412304056400, offset 192 00000400:00000200:2.0:1547251672.780751:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.10.184@o2ib10 00000400:00000200:2.0:1547251672.780755:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.10.184@o2ib10 local destination 00000400:00000200:2.0:1547251672.780757:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251672.780758:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251672.780760:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.0.10.184@o2ib10 c:[8, 8], s:[18, 19] 00000400:00000200:2.0:1547251672.780761:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.10.184@o2ib10 00000400:00000200:2.0:1547251672.780767:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.10.184@o2ib10(10.0.10.184@o2ib10:10.0.10.184@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251672.780769:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 312 bytes in 1 frags to 12345-10.0.10.184@o2ib10 00000800:00000200:2.0:1547251672.780772:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b971916c00] -> 10.0.10.184@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251672.780774:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b81d647a00] (20)++ 00000800:00000200:2.0:1547251672.780775:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:2.0:1547251672.780777:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b81d647a00] (22)-- 00000100:00000200:2.0:1547251672.780781:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:0.2:1547251672.780789:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:13.0:1547251672.780802:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)++ 00000800:00000200:13.0:1547251672.780807:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b81d647a00] (23)-- 00000400:00000200:13.0:1547251672.780810:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a5d8 00000400:00000200:13.0:1547251672.780811:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251672.780813:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.10.184@o2ib10: PUT: OK 00000800:00000200:15.0:1547251672.780815:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)-- 00000800:00000200:13.0:1547251672.780817:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (21)-- 00000800:00000200:8.2:1547251672.780921:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d02b200] (20)++ 00000800:00000200:15.0:1547251672.780936:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b200] (21)++ 00000800:00000200:13.0:1547251672.780952:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b200] (22)-- 00000800:00000200:15.0:1547251672.780953:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.10.184@o2ib10 00000400:00000200:15.0:1547251672.780956:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.10.184@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251672.780961:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.10.184@o2ib10 of length 296 into portal 26 MB=0x5c3934d100060 00000400:00000200:15.0:1547251672.780964:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.10.184@o2ib10 of length 296/296 into md 0xd5619 [1] + 0 00000400:00000200:15.0:1547251672.780966:0:12666:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d880 00000100:00000200:15.0:1547251672.780968:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:15.0:1547251672.780974:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d880 00000400:00000200:15.0:1547251672.780978:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251672.780980:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (21)++ 00000800:00000200:15.0:1547251672.780981:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (22)-- 00000800:00000200:15.0:1547251672.780982:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b200] (21)-- 00000100:00000200:2.0:1547251672.780990:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622412304056416 00010000:00000200:2.0:1547251672.781008:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692fea850 x1622412304056416/t0(0) o501->8534aa65-7ac8-198c-0718-3e6f83d57705@10.0.10.184@o2ib10:733/0 lens 296/272 e 0 to 0 dl 1547251678 ref 1 fl Interpret:/0/0 rc -2/0 00000100:00000200:2.0:1547251672.781014:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 240 bytes to portal 25, xid 1622412304056416, offset 192 00000400:00000200:2.0:1547251672.781016:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.10.184@o2ib10 00000400:00000200:2.0:1547251672.781019:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.10.184@o2ib10 local destination 00000400:00000200:2.0:1547251672.781021:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251672.781022:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251672.781023:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.0.10.184@o2ib10 c:[8, 8], s:[19, 19] 00000400:00000200:2.0:1547251672.781025:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.2.10.184@o2ib10 00000400:00000200:2.0:1547251672.781030:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.2.10.184@o2ib10(10.0.10.184@o2ib10:10.2.10.184@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251672.781032:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 240 bytes in 1 frags to 12345-10.2.10.184@o2ib10 00000800:00000200:2.0:1547251672.781037:0:12929:0:(o2iblnd_cb.c:1604:kiblnd_launch_tx()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (1)++ 00000800:00000200:2.0:1547251672.781040:0:12929:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:2.0:1547251672.781053:0:12929:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:2.0:1547251672.781054:0:12929:0:(o2iblnd_cb.c:1611:kiblnd_launch_tx()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000100:00000200:2.0:1547251672.781058:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:14.0:1547251672.781071:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:14.0:1547251672.781080:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:14.0:1547251672.781564:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:14.0:1547251672.781571:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b8246a0000 0x13e46a0000(0x13e46a0000) 00000800:00000200:14.0:1547251672.781572:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97a99502c000 0x55502c000(0x55502c000) 00000800:00000200:14.0:1547251672.781573:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba8fc43000 0x164fc43000(0x164fc43000) 00000800:00000200:14.0:1547251672.781574:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b17b3bf000 0xd3b3bf000(0xd3b3bf000) 00000800:00000200:14.0:1547251672.781575:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b17b1a7000 0xd3b1a7000(0xd3b1a7000) 00000800:00000200:14.0:1547251672.781576:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b17b2fe000 0xd3b2fe000(0xd3b2fe000) 00000800:00000200:14.0:1547251672.781577:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b824611000 0x13e4611000(0x13e4611000) 00000800:00000200:14.0:1547251672.781578:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b81d63e000 0x13dd63e000(0x13dd63e000) 00000800:00000200:14.0:1547251672.781579:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97a99520e000 0x55520e000(0x55520e000) 00000800:00000200:14.0:1547251672.781580:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b81d497000 0x13dd497000(0x13dd497000) 00000800:00000200:14.0:1547251672.781581:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b824664000 0x13e4664000(0x13e4664000) 00000800:00000200:14.0:1547251672.781582:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ae6a9f2000 0xa2a9f2000(0xa2a9f2000) 00000800:00000200:14.0:1547251672.781583:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b17b3c5000 0xd3b3c5000(0xd3b3c5000) 00000800:00000200:14.0:1547251672.781597:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ae6ab6b000 0xa2ab6b000(0xa2ab6b000) 00000800:00000200:14.0:1547251672.781598:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b82461b000 0x13e461b000(0x13e461b000) 00000800:00000200:14.0:1547251672.781600:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b81d5ad000 0x13dd5ad000(0x13dd5ad000) 00000800:00000200:14.0:1547251672.781601:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b81d787000 0x13dd787000(0x13dd787000) 00000800:00000200:14.0:1547251672.781602:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b907293000 0x14c7293000(0x14c7293000) 00000800:00000200:9.0:1547251672.782236:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782238:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782238:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782239:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782240:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782240:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782241:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782242:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782242:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782243:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782243:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782244:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782245:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782245:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782246:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782246:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782247:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782248:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782248:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782249:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782250:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782251:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782251:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782252:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782253:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782253:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782254:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782254:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782255:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782256:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782256:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782257:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782257:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782258:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:9.0:1547251672.782259:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:9.0:1547251672.782259:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000100:12.0:1547251672.782540:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251672.782543:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251672.782544:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251672.782589:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:12.0:1547251672.782602:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:13.0:1547251672.782614:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (19)++ 00000800:00000200:13.0:1547251672.782622:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (20)-- 00000800:00000200:13.0:1547251672.782623:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (19)-- 00000800:00000200:13.0:1547251672.782625:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (18)-- 00000800:00000200:8.2:1547251672.782727:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97babe8ace00] (17)++ 00000800:00000200:13.0:1547251672.782738:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (18)++ 00000800:00000200:13.0:1547251672.782743:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (19)-- 00000800:00000200:13.0:1547251672.782744:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (18)-- 00000800:00000200:13.0:1547251672.782745:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (17)++ 00000800:00000200:13.0:1547251672.782746:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (18)-- 00000800:00000200:13.0:1547251672.782746:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (17)-- 00000800:00000200:13.0:1547251672.782747:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (16)++ 00000800:00000200:13.0:1547251672.782748:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (17)-- 00000800:00000200:13.0:1547251672.782748:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (16)-- 00000800:00000200:13.0:1547251672.782749:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (15)++ 00000800:00000200:13.0:1547251672.782750:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (16)-- 00000800:00000200:13.0:1547251672.782751:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (15)-- 00000800:00000200:15.0:1547251672.782752:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (14)++ 00000800:00000200:15.0:1547251672.782759:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (15)-- 00000800:00000200:15.0:1547251672.782760:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (14)-- 00000800:00000200:15.0:1547251672.782761:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (13)++ 00000800:00000200:15.0:1547251672.782764:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (14)-- 00000800:00000200:15.0:1547251672.782765:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (13)-- 00000800:00000200:15.0:1547251672.782766:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (12)++ 00000800:00000200:15.0:1547251672.782767:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (13)-- 00000800:00000200:13.0:1547251672.782767:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (13)++ 00000800:00000200:15.0:1547251672.782768:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (13)-- 00000800:00000200:13.0:1547251672.782769:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (12)-- 00000800:00000200:15.0:1547251672.782770:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (12)++ 00000800:00000200:13.0:1547251672.782770:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (12)-- 00000800:00000200:15.0:1547251672.782771:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (11)-- 00000800:00000200:13.0:1547251672.782772:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (11)++ 00000800:00000200:15.0:1547251672.782773:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (11)-- 00000800:00000200:15.0:1547251672.782774:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (10)++ 00000800:00000200:13.0:1547251672.782774:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (10)-- 00000800:00000200:15.0:1547251672.782775:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (9)-- 00000800:00000200:13.0:1547251672.782775:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (10)-- 00000800:00000200:13.0:1547251672.782776:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (9)++ 00000800:00000200:15.0:1547251672.782777:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (9)-- 00000800:00000200:15.0:1547251672.782778:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (8)++ 00000800:00000200:13.0:1547251672.782778:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (8)-- 00000800:00000200:13.0:1547251672.782779:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (8)-- 00000800:00000200:15.0:1547251672.782780:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (7)-- 00000800:00000200:13.0:1547251672.782780:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (7)++ 00000800:00000200:15.0:1547251672.782781:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (7)-- 00000800:00000200:15.0:1547251672.782782:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (6)++ 00000800:00000200:13.0:1547251672.782782:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (6)-- 00000800:00000200:13.0:1547251672.782783:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (6)-- 00000800:00000200:15.0:1547251672.782784:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (5)-- 00000800:00000200:13.0:1547251672.782784:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (5)++ 00000800:00000200:15.0:1547251672.782785:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (5)-- 00000800:00000200:15.0:1547251672.782786:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97babe8ace00] (4)++ 00000800:00000200:13.0:1547251672.782786:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (4)-- 00000800:00000200:15.0:1547251672.782787:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97babe8ace00] (3)-- 00000800:00000200:13.0:1547251672.782787:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (4)-- 00000800:00000200:13.0:1547251672.782788:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (3)-- 00000800:00000200:15.0:1547251672.782789:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97babe8ace00] (1)-- 00000800:00000200:9.0:1547251672.782805:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.783479:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251672.783484:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.783495:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251672.783497:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:2.0:1547251672.783503:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:2.0:1547251672.783512:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:2.0:1547251672.784035:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:2.0:1547251672.784044:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b81d52c000 0x13dd52c000(0x13dd52c000) 00000800:00000200:2.0:1547251672.784045:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba941f3000 0x16541f3000(0x16541f3000) 00000800:00000200:2.0:1547251672.784046:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97bae04ed000 0x16a04ed000(0x16a04ed000) 00000800:00000200:2.0:1547251672.784047:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b17b0b9000 0xd3b0b9000(0xd3b0b9000) 00000800:00000200:2.0:1547251672.784048:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b81d517000 0x13dd517000(0x13dd517000) 00000800:00000200:2.0:1547251672.784049:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97a9952e3000 0x5552e3000(0x5552e3000) 00000800:00000200:2.0:1547251672.784050:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ae3782d000 0x9f782d000(0x9f782d000) 00000800:00000200:2.0:1547251672.784051:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97a65d0ea000 0x21d0ea000(0x21d0ea000) 00000800:00000200:2.0:1547251672.784052:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ae6a9fc000 0xa2a9fc000(0xa2a9fc000) 00000800:00000200:2.0:1547251672.784053:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ae5d2c5000 0xa1d2c5000(0xa1d2c5000) 00000800:00000200:2.0:1547251672.784053:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b81d76f000 0x13dd76f000(0x13dd76f000) 00000800:00000200:2.0:1547251672.784054:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ae6ab19000 0xa2ab19000(0xa2ab19000) 00000800:00000200:2.0:1547251672.784055:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ae3787e000 0x9f787e000(0x9f787e000) 00000800:00000200:2.0:1547251672.784056:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97a995224000 0x555224000(0x555224000) 00000800:00000200:2.0:1547251672.784057:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b17b399000 0xd3b399000(0xd3b399000) 00000800:00000200:2.0:1547251672.784058:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b17b1b1000 0xd3b1b1000(0xd3b1b1000) 00000800:00000200:2.0:1547251672.784059:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ae6aaa5000 0xa2aaa5000(0xa2aaa5000) 00000800:00000200:2.0:1547251672.784059:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ae6aa3c000 0xa2aa3c000(0xa2aa3c000) 00000800:00000200:9.0:1547251672.784735:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784737:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784738:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784739:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784739:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784740:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784741:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784741:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784742:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784743:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784743:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784744:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784744:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784745:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784746:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784746:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784748:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784748:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784749:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784749:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784750:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784751:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784751:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784752:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784752:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784753:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784754:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784754:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784755:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784755:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784756:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784757:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784757:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784758:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:9.0:1547251672.784759:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b600] (19)++ 00000800:00000200:9.0:1547251672.784759:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000100:12.0:1547251672.785039:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251672.785042:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251672.785044:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251672.785081:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b82583b600] (19)++ 00000800:00000200:13.0:1547251672.785106:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (20)++ 00000800:00000200:12.0:1547251672.785106:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97b82583b600] (20)-- 00000800:00000200:13.0:1547251672.785114:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (20)-- 00000800:00000200:13.0:1547251672.785116:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (19)-- 00000800:00000200:13.0:1547251672.785117:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (18)-- 00000800:00000200:8.2:1547251672.785220:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b82583b600] (17)++ 00000800:00000200:13.0:1547251672.785231:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (18)++ 00000800:00000200:13.0:1547251672.785236:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (19)-- 00000800:00000200:13.0:1547251672.785237:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (18)-- 00000800:00000200:13.0:1547251672.785237:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (17)++ 00000800:00000200:13.0:1547251672.785238:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (18)-- 00000800:00000200:13.0:1547251672.785240:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (17)-- 00000800:00000200:13.0:1547251672.785241:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (16)++ 00000800:00000200:13.0:1547251672.785243:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (17)-- 00000800:00000200:13.0:1547251672.785244:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (16)-- 00000800:00000200:15.0:1547251672.785245:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (15)++ 00000800:00000200:15.0:1547251672.785252:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (16)-- 00000800:00000200:15.0:1547251672.785253:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (15)-- 00000800:00000200:15.0:1547251672.785254:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (14)++ 00000800:00000200:15.0:1547251672.785257:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (15)-- 00000800:00000200:15.0:1547251672.785257:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (14)-- 00000800:00000200:15.0:1547251672.785258:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (13)++ 00000800:00000200:15.0:1547251672.785259:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (14)-- 00000800:00000200:13.0:1547251672.785260:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (14)++ 00000800:00000200:15.0:1547251672.785261:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (14)-- 00000800:00000200:15.0:1547251672.785263:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (13)++ 00000800:00000200:13.0:1547251672.785263:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (13)-- 00000800:00000200:13.0:1547251672.785264:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (13)-- 00000800:00000200:15.0:1547251672.785265:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (12)-- 00000800:00000200:15.0:1547251672.785266:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (12)-- 00000800:00000200:13.0:1547251672.785266:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (12)++ 00000800:00000200:13.0:1547251672.785267:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (11)-- 00000800:00000200:15.0:1547251672.785268:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (11)++ 00000800:00000200:15.0:1547251672.785269:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (10)-- 00000800:00000200:13.0:1547251672.785269:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (11)-- 00000800:00000200:13.0:1547251672.785270:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (10)++ 00000800:00000200:15.0:1547251672.785271:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (10)-- 00000800:00000200:13.0:1547251672.785271:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (9)-- 00000800:00000200:15.0:1547251672.785272:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (9)++ 00000800:00000200:15.0:1547251672.785273:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (8)-- 00000800:00000200:13.0:1547251672.785273:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (9)-- 00000800:00000200:13.0:1547251672.785274:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (8)++ 00000800:00000200:15.0:1547251672.785275:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (8)-- 00000800:00000200:13.0:1547251672.785275:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (7)-- 00000800:00000200:15.0:1547251672.785276:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (7)++ 00000800:00000200:15.0:1547251672.785277:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (6)-- 00000800:00000200:13.0:1547251672.785277:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (7)-- 00000800:00000200:15.0:1547251672.785278:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (6)-- 00000800:00000200:13.0:1547251672.785278:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (6)++ 00000800:00000200:13.0:1547251672.785279:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (5)-- 00000800:00000200:15.0:1547251672.785280:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (5)++ 00000800:00000200:13.0:1547251672.785280:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (5)-- 00000800:00000200:15.0:1547251672.785281:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (4)-- 00000800:00000200:13.0:1547251672.785281:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b600] (4)++ 00000800:00000200:15.0:1547251672.785282:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (4)-- 00000800:00000200:13.0:1547251672.785283:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b600] (3)-- 00000800:00000200:15.0:1547251672.785284:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (3)-- 00000800:00000200:13.0:1547251672.785285:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b600] (1)-- 00000800:00000200:9.0:1547251672.785297:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.785959:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251672.785964:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.785975:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251672.785977:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251672.785983:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251672.785992:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251672.786491:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251672.786498:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba961ae000 0x16561ae000(0x16561ae000) 00000800:00000200:10.0:1547251672.786500:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b80cd1b000 0x13ccd1b000(0x13ccd1b000) 00000800:00000200:10.0:1547251672.786501:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97baa1dbf000 0x1661dbf000(0x1661dbf000) 00000800:00000200:10.0:1547251672.786502:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97badc8bf000 0x169c8bf000(0x169c8bf000) 00000800:00000200:10.0:1547251672.786503:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ba9bf33000 0x165bf33000(0x165bf33000) 00000800:00000200:10.0:1547251672.786503:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97baa0e2c000 0x1660e2c000(0x1660e2c000) 00000800:00000200:10.0:1547251672.786504:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97a72db0b000 0x2edb0b000(0x2edb0b000) 00000800:00000200:10.0:1547251672.786505:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba8f7b5000 0x164f7b5000(0x164f7b5000) 00000800:00000200:10.0:1547251672.786506:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97bade7f1000 0x169e7f1000(0x169e7f1000) 00000800:00000200:10.0:1547251672.786507:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97badb94f000 0x169b94f000(0x169b94f000) 00000800:00000200:10.0:1547251672.786508:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b158afd000 0xd18afd000(0xd18afd000) 00000800:00000200:10.0:1547251672.786508:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97badc907000 0x169c907000(0x169c907000) 00000800:00000200:10.0:1547251672.786509:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b073a9d000 0xc33a9d000(0xc33a9d000) 00000800:00000200:10.0:1547251672.786510:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b6fde3e000 0x12bde3e000(0x12bde3e000) 00000800:00000200:10.0:1547251672.786511:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba92fc4000 0x1652fc4000(0x1652fc4000) 00000800:00000200:10.0:1547251672.786512:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97badc9ea000 0x169c9ea000(0x169c9ea000) 00000800:00000200:10.0:1547251672.786513:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b9073ab000 0x14c73ab000(0x14c73ab000) 00000800:00000200:10.0:1547251672.786513:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba95590000 0x1655590000(0x1655590000) 00000800:00000200:9.0:1547251672.787107:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787108:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787109:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787110:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787110:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787111:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787112:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787112:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787113:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787113:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787114:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787115:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787116:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787116:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787117:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787118:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787118:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787119:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787119:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787120:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787120:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787121:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787122:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787122:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787123:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787123:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787124:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787125:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787125:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787126:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787126:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787127:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787128:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787128:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251672.787129:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251672.787129:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000100:12.0:1547251672.787394:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251672.787397:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251672.787399:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251672.787449:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:12.0:1547251672.787466:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:15.0:1547251672.787509:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:15.0:1547251672.787517:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:15.0:1547251672.787518:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (19)-- 00000800:00000200:15.0:1547251672.787519:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (18)-- 00000800:00000200:8.2:1547251672.787532:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e800] (17)++ 00000800:00000200:15.0:1547251672.787545:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (18)++ 00000800:00000200:15.0:1547251672.787550:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (19)-- 00000800:00000200:15.0:1547251672.787550:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (18)-- 00000800:00000200:15.0:1547251672.787551:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (17)++ 00000800:00000200:15.0:1547251672.787552:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (18)-- 00000800:00000200:15.0:1547251672.787553:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (17)-- 00000800:00000200:15.0:1547251672.787553:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (16)++ 00000800:00000200:15.0:1547251672.787554:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (17)-- 00000800:00000200:15.0:1547251672.787555:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (16)-- 00000800:00000200:15.0:1547251672.787556:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (15)++ 00000800:00000200:15.0:1547251672.787557:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (16)-- 00000800:00000200:15.0:1547251672.787558:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (15)-- 00000800:00000200:13.0:1547251672.787559:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (14)++ 00000800:00000200:13.0:1547251672.787565:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (15)-- 00000800:00000200:13.0:1547251672.787566:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (14)-- 00000800:00000200:13.0:1547251672.787567:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (13)++ 00000800:00000200:13.0:1547251672.787569:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (14)-- 00000800:00000200:13.0:1547251672.787570:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (13)-- 00000800:00000200:13.0:1547251672.787571:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (12)++ 00000800:00000200:15.0:1547251672.787572:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (13)++ 00000800:00000200:13.0:1547251672.787572:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (13)-- 00000800:00000200:13.0:1547251672.787573:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (13)-- 00000800:00000200:15.0:1547251672.787574:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (12)-- 00000800:00000200:15.0:1547251672.787575:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (12)-- 00000800:00000200:13.0:1547251672.787575:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (12)++ 00000800:00000200:13.0:1547251672.787576:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (11)-- 00000800:00000200:15.0:1547251672.787577:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (11)++ 00000800:00000200:13.0:1547251672.787578:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (11)-- 00000800:00000200:15.0:1547251672.787579:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (10)-- 00000800:00000200:13.0:1547251672.787579:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (10)++ 00000800:00000200:13.0:1547251672.787580:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (11)-- 00000800:00000200:13.0:1547251672.787580:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (10)-- 00000800:00000200:15.0:1547251672.787582:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (8)-- 00000800:00000200:13.0:1547251672.787585:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (7)++ 00000800:00000200:15.0:1547251672.787595:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (8)++ 00000800:00000200:15.0:1547251672.787596:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (9)-- 00000800:00000200:15.0:1547251672.787597:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (8)-- 00000800:00000200:15.0:1547251672.787597:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (7)++ 00000800:00000200:15.0:1547251672.787598:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (8)-- 00000800:00000200:15.0:1547251672.787599:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (7)-- 00000800:00000200:15.0:1547251672.787599:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (6)++ 00000800:00000200:15.0:1547251672.787600:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (7)-- 00000800:00000200:15.0:1547251672.787601:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (6)-- 00000800:00000200:15.0:1547251672.787601:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (5)++ 00000800:00000200:15.0:1547251672.787602:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (6)-- 00000800:00000200:15.0:1547251672.787603:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (5)-- 00000800:00000200:15.0:1547251672.787603:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (4)++ 00000800:00000200:15.0:1547251672.787604:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (5)-- 00000800:00000200:15.0:1547251672.787605:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (4)-- 00000800:00000200:15.0:1547251672.787606:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (3)-- 00000800:00000200:13.0:1547251672.787610:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (2)-- 00000800:00000200:13.0:1547251672.787611:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (1)-- 00000800:00000200:9.0:1547251672.787620:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.788477:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251672.788482:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.788491:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251672.788492:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251672.788499:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251672.788507:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251672.788804:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251672.788813:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba927e7000 0x16527e7000(0x16527e7000) 00000800:00000200:10.0:1547251672.788814:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba8f6ca000 0x164f6ca000(0x164f6ca000) 00000800:00000200:10.0:1547251672.788815:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97badcaf8000 0x169caf8000(0x169caf8000) 00000800:00000200:10.0:1547251672.788816:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ba9b621000 0x165b621000(0x165b621000) 00000800:00000200:10.0:1547251672.788817:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b62c426000 0x11ec426000(0x11ec426000) 00000800:00000200:10.0:1547251672.788818:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b5872c8000 0x11472c8000(0x11472c8000) 00000800:00000200:10.0:1547251672.788819:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97badc80b000 0x169c80b000(0x169c80b000) 00000800:00000200:10.0:1547251672.788820:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ad4c2d6000 0x90c2d6000(0x90c2d6000) 00000800:00000200:10.0:1547251672.788821:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ba997d1000 0x16597d1000(0x16597d1000) 00000800:00000200:10.0:1547251672.788822:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba9ef48000 0x165ef48000(0x165ef48000) 00000800:00000200:10.0:1547251672.788823:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97baa13a2000 0x16613a2000(0x16613a2000) 00000800:00000200:10.0:1547251672.788823:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba9d3c1000 0x165d3c1000(0x165d3c1000) 00000800:00000200:10.0:1547251672.788824:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba996e3000 0x16596e3000(0x16596e3000) 00000800:00000200:10.0:1547251672.788825:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97bad6ae0000 0x1696ae0000(0x1696ae0000) 00000800:00000200:10.0:1547251672.788826:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97a589ecb000 0x149ecb000(0x149ecb000) 00000800:00000200:10.0:1547251672.788827:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97baa0f30000 0x1660f30000(0x1660f30000) 00000800:00000200:10.0:1547251672.788828:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97baddebc000 0x169debc000(0x169debc000) 00000800:00000200:10.0:1547251672.788829:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba9ba4b000 0x165ba4b000(0x165ba4b000) 00000800:00000200:9.0:1547251672.789362:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789363:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789364:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789365:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789365:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789366:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789367:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789367:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789368:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789369:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789370:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789370:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789371:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789372:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789372:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789373:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789373:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789374:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789375:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789375:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789376:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789376:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789377:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789378:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789378:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789379:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789379:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789380:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789381:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789381:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789382:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789382:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789383:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789384:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251672.789384:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251672.789385:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000100:12.0:1547251672.789648:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251672.789651:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251672.789653:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251672.789695:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:12.0:1547251672.789702:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:15.0:1547251672.789746:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:15.0:1547251672.789755:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:15.0:1547251672.789755:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (19)-- 00000800:00000200:15.0:1547251672.789757:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (18)-- 00000800:00000200:8.2:1547251672.789832:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e600] (17)++ 00000800:00000200:15.0:1547251672.789844:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (18)++ 00000800:00000200:15.0:1547251672.789848:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (19)-- 00000800:00000200:15.0:1547251672.789849:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (18)-- 00000800:00000200:15.0:1547251672.789850:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (17)++ 00000800:00000200:15.0:1547251672.789851:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (18)-- 00000800:00000200:15.0:1547251672.789851:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (17)-- 00000800:00000200:15.0:1547251672.789852:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (16)++ 00000800:00000200:15.0:1547251672.789853:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (17)-- 00000800:00000200:15.0:1547251672.789853:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (16)-- 00000800:00000200:15.0:1547251672.789854:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (15)++ 00000800:00000200:15.0:1547251672.789858:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (16)-- 00000800:00000200:15.0:1547251672.789858:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (15)-- 00000800:00000200:15.0:1547251672.789859:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (14)++ 00000800:00000200:15.0:1547251672.789860:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (15)-- 00000800:00000200:15.0:1547251672.789861:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (14)-- 00000800:00000200:13.0:1547251672.789862:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (13)++ 00000800:00000200:13.0:1547251672.789867:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (14)-- 00000800:00000200:13.0:1547251672.789867:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (13)-- 00000800:00000200:13.0:1547251672.789868:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (12)++ 00000800:00000200:13.0:1547251672.789871:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (13)-- 00000800:00000200:13.0:1547251672.789872:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (12)-- 00000800:00000200:13.0:1547251672.789873:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (11)++ 00000800:00000200:15.0:1547251672.789874:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (12)++ 00000800:00000200:13.0:1547251672.789874:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (12)-- 00000800:00000200:13.0:1547251672.789877:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (12)-- 00000800:00000200:15.0:1547251672.789878:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (11)-- 00000800:00000200:13.0:1547251672.789878:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (11)++ 00000800:00000200:15.0:1547251672.789879:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (11)-- 00000800:00000200:15.0:1547251672.789880:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (10)++ 00000800:00000200:13.0:1547251672.789880:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (10)-- 00000800:00000200:13.0:1547251672.789881:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (10)-- 00000800:00000200:15.0:1547251672.789882:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (9)-- 00000800:00000200:13.0:1547251672.789882:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (9)++ 00000800:00000200:15.0:1547251672.789883:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (9)-- 00000800:00000200:15.0:1547251672.789884:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (8)++ 00000800:00000200:13.0:1547251672.789884:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (8)-- 00000800:00000200:13.0:1547251672.789885:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (8)-- 00000800:00000200:15.0:1547251672.789886:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (7)-- 00000800:00000200:13.0:1547251672.789886:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (7)++ 00000800:00000200:15.0:1547251672.789887:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (7)-- 00000800:00000200:15.0:1547251672.789888:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (6)++ 00000800:00000200:13.0:1547251672.789888:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (6)-- 00000800:00000200:13.0:1547251672.789889:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (6)-- 00000800:00000200:15.0:1547251672.789890:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (5)-- 00000800:00000200:13.0:1547251672.789890:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (5)++ 00000800:00000200:15.0:1547251672.789891:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (5)-- 00000800:00000200:13.0:1547251672.789891:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (4)-- 00000800:00000200:15.0:1547251672.789892:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (4)++ 00000800:00000200:15.0:1547251672.789893:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (3)-- 00000800:00000200:13.0:1547251672.789893:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (4)-- 00000800:00000200:13.0:1547251672.789894:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (3)-- 00000800:00000200:15.0:1547251672.789895:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (1)-- 00000800:00000200:9.0:1547251672.789906:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.790676:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251672.790681:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.790689:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251672.790691:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251672.790699:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251672.790706:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251672.791099:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251672.791108:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97bad6a8c000 0x1696a8c000(0x1696a8c000) 00000800:00000200:10.0:1547251672.791109:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba9ce49000 0x165ce49000(0x165ce49000) 00000800:00000200:10.0:1547251672.791110:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97bae1cfd000 0x16a1cfd000(0x16a1cfd000) 00000800:00000200:10.0:1547251672.791111:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ba90fac000 0x1650fac000(0x1650fac000) 00000800:00000200:10.0:1547251672.791112:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97a601aa3000 0x1c1aa3000(0x1c1aa3000) 00000800:00000200:10.0:1547251672.791113:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba95c64000 0x1655c64000(0x1655c64000) 00000800:00000200:10.0:1547251672.791114:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ac1b2d3000 0x7db2d3000(0x7db2d3000) 00000800:00000200:10.0:1547251672.791115:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b4f05eb000 0x10b05eb000(0x10b05eb000) 00000800:00000200:10.0:1547251672.791115:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b17b185000 0xd3b185000(0xd3b185000) 00000800:00000200:10.0:1547251672.791116:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b82459e000 0x13e459e000(0x13e459e000) 00000800:00000200:10.0:1547251672.791117:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b82459f000 0x13e459f000(0x13e459f000) 00000800:00000200:10.0:1547251672.791118:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97baa134a000 0x166134a000(0x166134a000) 00000800:00000200:10.0:1547251672.791119:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97baa134b000 0x166134b000(0x166134b000) 00000800:00000200:10.0:1547251672.791120:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba90fc4000 0x1650fc4000(0x1650fc4000) 00000800:00000200:10.0:1547251672.791121:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba90fc5000 0x1650fc5000(0x1650fc5000) 00000800:00000200:10.0:1547251672.791122:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ae6aaa0000 0xa2aaa0000(0xa2aaa0000) 00000800:00000200:10.0:1547251672.791123:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ae6aa9f000 0xa2aa9f000(0xa2aa9f000) 00000800:00000200:10.0:1547251672.791124:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b8247cb000 0x13e47cb000(0x13e47cb000) 00000800:00000200:9.0:1547251672.791712:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791713:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791714:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791714:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791715:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791716:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791717:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791717:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791718:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791719:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791719:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791720:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791720:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791721:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791722:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791722:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791723:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791724:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791724:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791725:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791725:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791726:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791727:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791727:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791728:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791728:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791729:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791730:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791730:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791731:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791731:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791732:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791733:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791733:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:9.0:1547251672.791734:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:9.0:1547251672.791734:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000100:12.0:1547251672.792005:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251672.792007:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251672.792009:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251672.792053:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:12.0:1547251672.792071:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:13.0:1547251672.792074:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (19)++ 00000800:00000200:13.0:1547251672.792080:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (20)-- 00000800:00000200:13.0:1547251672.792081:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (19)-- 00000800:00000200:13.0:1547251672.792083:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (18)-- 00000800:00000200:8.2:1547251672.792193:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f600] (17)++ 00000800:00000200:13.0:1547251672.792205:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (18)++ 00000800:00000200:13.0:1547251672.792209:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (19)-- 00000800:00000200:13.0:1547251672.792210:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (18)-- 00000800:00000200:13.0:1547251672.792211:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (17)++ 00000800:00000200:13.0:1547251672.792211:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (18)-- 00000800:00000200:13.0:1547251672.792212:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (17)-- 00000800:00000200:13.0:1547251672.792213:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (16)++ 00000800:00000200:13.0:1547251672.792214:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (17)-- 00000800:00000200:13.0:1547251672.792215:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (16)-- 00000800:00000200:13.0:1547251672.792216:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (15)++ 00000800:00000200:13.0:1547251672.792219:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (16)-- 00000800:00000200:13.0:1547251672.792220:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (15)-- 00000800:00000200:13.0:1547251672.792220:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (14)++ 00000800:00000200:13.0:1547251672.792222:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (15)-- 00000800:00000200:13.0:1547251672.792222:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (14)-- 00000800:00000200:15.0:1547251672.792224:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (13)++ 00000800:00000200:15.0:1547251672.792231:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (14)-- 00000800:00000200:15.0:1547251672.792231:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (13)-- 00000800:00000200:15.0:1547251672.792233:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (12)++ 00000800:00000200:15.0:1547251672.792236:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (13)-- 00000800:00000200:15.0:1547251672.792236:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (12)-- 00000800:00000200:15.0:1547251672.792237:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (11)++ 00000800:00000200:15.0:1547251672.792238:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (12)-- 00000800:00000200:15.0:1547251672.792239:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (12)-- 00000800:00000200:13.0:1547251672.792239:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (12)++ 00000800:00000200:13.0:1547251672.792240:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (11)-- 00000800:00000200:15.0:1547251672.792241:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (11)++ 00000800:00000200:13.0:1547251672.792241:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (11)-- 00000800:00000200:15.0:1547251672.792242:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (10)-- 00000800:00000200:13.0:1547251672.792243:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (10)++ 00000800:00000200:15.0:1547251672.792244:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (10)-- 00000800:00000200:13.0:1547251672.792244:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (9)-- 00000800:00000200:15.0:1547251672.792245:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (9)++ 00000800:00000200:15.0:1547251672.792246:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (8)-- 00000800:00000200:13.0:1547251672.792246:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (9)-- 00000800:00000200:15.0:1547251672.792247:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (8)-- 00000800:00000200:13.0:1547251672.792247:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (8)++ 00000800:00000200:13.0:1547251672.792248:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (7)-- 00000800:00000200:15.0:1547251672.792249:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (7)++ 00000800:00000200:13.0:1547251672.792249:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (7)-- 00000800:00000200:15.0:1547251672.792250:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (6)-- 00000800:00000200:15.0:1547251672.792251:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (6)-- 00000800:00000200:13.0:1547251672.792251:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (6)++ 00000800:00000200:13.0:1547251672.792252:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (5)-- 00000800:00000200:15.0:1547251672.792253:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (5)++ 00000800:00000200:13.0:1547251672.792254:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (5)-- 00000800:00000200:15.0:1547251672.792255:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (4)-- 00000800:00000200:13.0:1547251672.792255:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f600] (4)++ 00000800:00000200:15.0:1547251672.792256:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (4)-- 00000800:00000200:15.0:1547251672.792257:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (3)-- 00000800:00000200:13.0:1547251672.792257:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f600] (3)-- 00000800:00000200:13.0:1547251672.792258:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f600] (1)-- 00000800:00000200:9.0:1547251672.792270:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.793007:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251672.793012:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.793021:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251672.793022:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251672.793029:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251672.793037:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251672.793289:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251672.793295:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b17b3df000 0xd3b3df000(0xd3b3df000) 00000800:00000200:10.0:1547251672.793297:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97a65d357000 0x21d357000(0x21d357000) 00000800:00000200:10.0:1547251672.793298:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ae6a97a000 0xa2a97a000(0xa2a97a000) 00000800:00000200:10.0:1547251672.793299:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b81d767000 0x13dd767000(0x13dd767000) 00000800:00000200:10.0:1547251672.793300:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ae6a9aa000 0xa2a9aa000(0xa2a9aa000) 00000800:00000200:10.0:1547251672.793301:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ae6ab74000 0xa2ab74000(0xa2ab74000) 00000800:00000200:10.0:1547251672.793301:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b9072d3000 0x14c72d3000(0x14c72d3000) 00000800:00000200:10.0:1547251672.793302:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b81d638000 0x13dd638000(0x13dd638000) 00000800:00000200:10.0:1547251672.793303:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b81d689000 0x13dd689000(0x13dd689000) 00000800:00000200:10.0:1547251672.793304:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b81d40b000 0x13dd40b000(0x13dd40b000) 00000800:00000200:10.0:1547251672.793305:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ae6ab11000 0xa2ab11000(0xa2ab11000) 00000800:00000200:10.0:1547251672.793306:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b907293000 0x14c7293000(0x14c7293000) 00000800:00000200:10.0:1547251672.793307:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b81d787000 0x13dd787000(0x13dd787000) 00000800:00000200:10.0:1547251672.793307:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b81d5ad000 0x13dd5ad000(0x13dd5ad000) 00000800:00000200:10.0:1547251672.793308:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b82461b000 0x13e461b000(0x13e461b000) 00000800:00000200:10.0:1547251672.793309:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ae6ab6b000 0xa2ab6b000(0xa2ab6b000) 00000800:00000200:10.0:1547251672.793310:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b17b3c5000 0xd3b3c5000(0xd3b3c5000) 00000800:00000200:10.0:1547251672.793311:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ae6a9f2000 0xa2a9f2000(0xa2a9f2000) 00000800:00000200:9.0:1547251672.793847:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793848:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793849:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793849:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793850:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793851:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793851:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793852:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793852:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793853:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793854:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793854:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793855:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793855:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793856:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793857:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793857:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793858:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793858:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793859:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793860:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793860:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793861:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793861:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793862:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793863:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793863:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793864:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793864:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793865:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793866:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793866:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793867:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793867:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:9.0:1547251672.793868:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:9.0:1547251672.793869:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000100:12.0:1547251672.794206:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251672.794209:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251672.794211:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:12.0:1547251672.794260:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c200] (19)-- 00000800:00000200:8.2:1547251672.794267:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c200] (18)++ 00000800:00000200:15.0:1547251672.794321:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (19)++ 00000800:00000200:15.0:1547251672.794330:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (20)-- 00000800:00000200:15.0:1547251672.794331:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (19)-- 00000800:00000200:15.0:1547251672.794333:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (18)-- 00000800:00000200:8.2:1547251672.794403:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c200] (17)++ 00000800:00000200:15.0:1547251672.794415:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (18)++ 00000800:00000200:15.0:1547251672.794420:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (19)-- 00000800:00000200:15.0:1547251672.794421:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (18)-- 00000800:00000200:15.0:1547251672.794422:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (17)++ 00000800:00000200:15.0:1547251672.794422:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (18)-- 00000800:00000200:15.0:1547251672.794423:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (17)-- 00000800:00000200:15.0:1547251672.794424:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (16)++ 00000800:00000200:15.0:1547251672.794425:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (17)-- 00000800:00000200:15.0:1547251672.794425:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (16)-- 00000800:00000200:15.0:1547251672.794426:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (15)++ 00000800:00000200:15.0:1547251672.794429:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (16)-- 00000800:00000200:15.0:1547251672.794430:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (15)-- 00000800:00000200:15.0:1547251672.794431:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (14)++ 00000800:00000200:15.0:1547251672.794432:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (15)-- 00000800:00000200:15.0:1547251672.794432:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (14)-- 00000800:00000200:13.0:1547251672.794433:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (13)++ 00000800:00000200:13.0:1547251672.794439:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (14)-- 00000800:00000200:13.0:1547251672.794440:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (13)-- 00000800:00000200:13.0:1547251672.794441:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (12)++ 00000800:00000200:13.0:1547251672.794444:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (13)-- 00000800:00000200:13.0:1547251672.794445:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (12)-- 00000800:00000200:13.0:1547251672.794446:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (11)++ 00000800:00000200:15.0:1547251672.794447:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (12)++ 00000800:00000200:13.0:1547251672.794447:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (12)-- 00000800:00000200:13.0:1547251672.794448:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (12)-- 00000800:00000200:15.0:1547251672.794449:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (11)-- 00000800:00000200:13.0:1547251672.794449:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (11)++ 00000800:00000200:15.0:1547251672.794450:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (11)-- 00000800:00000200:15.0:1547251672.794451:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (10)++ 00000800:00000200:13.0:1547251672.794451:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (10)-- 00000800:00000200:13.0:1547251672.794452:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (10)-- 00000800:00000200:15.0:1547251672.794453:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (9)-- 00000800:00000200:13.0:1547251672.794453:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (9)++ 00000800:00000200:15.0:1547251672.794454:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (9)-- 00000800:00000200:15.0:1547251672.794455:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (8)++ 00000800:00000200:13.0:1547251672.794455:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (8)-- 00000800:00000200:13.0:1547251672.794456:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (8)-- 00000800:00000200:15.0:1547251672.794458:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (7)-- 00000800:00000200:13.0:1547251672.794458:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (7)++ 00000800:00000200:15.0:1547251672.794459:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (7)-- 00000800:00000200:15.0:1547251672.794460:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (6)++ 00000800:00000200:13.0:1547251672.794460:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (6)-- 00000800:00000200:13.0:1547251672.794461:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (6)-- 00000800:00000200:15.0:1547251672.794462:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (5)-- 00000800:00000200:13.0:1547251672.794463:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (5)++ 00000800:00000200:15.0:1547251672.794464:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (5)-- 00000800:00000200:13.0:1547251672.794464:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (4)-- 00000800:00000200:15.0:1547251672.794465:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c200] (4)++ 00000800:00000200:13.0:1547251672.794466:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (4)-- 00000800:00000200:15.0:1547251672.794467:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c200] (3)-- 00000800:00000200:13.0:1547251672.794467:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (3)-- 00000800:00000200:15.0:1547251672.794468:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c200] (1)-- 00000800:00000200:9.0:1547251672.794480:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.795276:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251672.795280:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.795288:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251672.795289:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251672.795296:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251672.795305:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251672.795549:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251672.795557:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b81d497000 0x13dd497000(0x13dd497000) 00000800:00000200:10.0:1547251672.795558:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97a99520e000 0x55520e000(0x55520e000) 00000800:00000200:10.0:1547251672.795560:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b81d63e000 0x13dd63e000(0x13dd63e000) 00000800:00000200:10.0:1547251672.795561:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b824611000 0x13e4611000(0x13e4611000) 00000800:00000200:10.0:1547251672.795562:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b17b2fe000 0xd3b2fe000(0xd3b2fe000) 00000800:00000200:10.0:1547251672.795563:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b17b1a7000 0xd3b1a7000(0xd3b1a7000) 00000800:00000200:10.0:1547251672.795563:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b17b3bf000 0xd3b3bf000(0xd3b3bf000) 00000800:00000200:10.0:1547251672.795564:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97bade7f1000 0x169e7f1000(0x169e7f1000) 00000800:00000200:10.0:1547251672.795565:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97badb94f000 0x169b94f000(0x169b94f000) 00000800:00000200:10.0:1547251672.795566:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b158afd000 0xd18afd000(0xd18afd000) 00000800:00000200:10.0:1547251672.795567:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97badc907000 0x169c907000(0x169c907000) 00000800:00000200:10.0:1547251672.795568:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b073a9d000 0xc33a9d000(0xc33a9d000) 00000800:00000200:10.0:1547251672.795569:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b6fde3e000 0x12bde3e000(0x12bde3e000) 00000800:00000200:10.0:1547251672.795569:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba92fc4000 0x1652fc4000(0x1652fc4000) 00000800:00000200:10.0:1547251672.795570:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97badc9ea000 0x169c9ea000(0x169c9ea000) 00000800:00000200:10.0:1547251672.795571:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b9073ab000 0x14c73ab000(0x14c73ab000) 00000800:00000200:10.0:1547251672.795572:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ba95590000 0x1655590000(0x1655590000) 00000800:00000200:10.0:1547251672.795573:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ba941f3000 0x16541f3000(0x16541f3000) 00000800:00000200:9.0:1547251672.796123:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796125:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796125:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796126:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796127:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796127:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796128:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796129:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796129:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796130:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796130:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796131:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796132:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796132:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796133:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796133:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796134:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796135:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796135:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796136:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796136:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796137:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796138:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796138:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796139:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796139:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796140:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796141:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796141:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796142:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796142:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796143:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796144:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796145:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:9.0:1547251672.796145:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:9.0:1547251672.796146:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000100:12.0:1547251672.796406:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251672.796409:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251672.796411:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251672.796462:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:12.0:1547251672.796465:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:13.0:1547251672.796486:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (19)++ 00000800:00000200:13.0:1547251672.796494:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (20)-- 00000800:00000200:13.0:1547251672.796495:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (19)-- 00000800:00000200:13.0:1547251672.796497:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (18)-- 00000800:00000200:8.2:1547251672.796597:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f800] (17)++ 00000800:00000200:13.0:1547251672.796606:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (18)++ 00000800:00000200:13.0:1547251672.796612:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (19)-- 00000800:00000200:13.0:1547251672.796613:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (18)-- 00000800:00000200:13.0:1547251672.796614:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (17)++ 00000800:00000200:13.0:1547251672.796616:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (18)-- 00000800:00000200:13.0:1547251672.796617:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (17)-- 00000800:00000200:13.0:1547251672.796618:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (16)++ 00000800:00000200:13.0:1547251672.796620:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (17)-- 00000800:00000200:13.0:1547251672.796621:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (16)-- 00000800:00000200:15.0:1547251672.796623:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (15)++ 00000800:00000200:15.0:1547251672.796629:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (16)-- 00000800:00000200:15.0:1547251672.796630:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (15)-- 00000800:00000200:15.0:1547251672.796631:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (14)++ 00000800:00000200:15.0:1547251672.796634:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (15)-- 00000800:00000200:13.0:1547251672.796634:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (15)++ 00000800:00000200:15.0:1547251672.796635:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (15)-- 00000800:00000200:15.0:1547251672.796636:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (14)++ 00000800:00000200:13.0:1547251672.796636:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (14)-- 00000800:00000200:15.0:1547251672.796637:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (14)-- 00000800:00000200:13.0:1547251672.796637:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (14)-- 00000800:00000200:15.0:1547251672.796639:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (12)-- 00000800:00000200:13.0:1547251672.796640:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (11)++ 00000800:00000200:13.0:1547251672.796644:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (12)-- 00000800:00000200:13.0:1547251672.796646:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (11)-- 00000800:00000200:13.0:1547251672.796647:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (10)++ 00000800:00000200:13.0:1547251672.796651:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (11)-- 00000800:00000200:13.0:1547251672.796652:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (10)-- 00000800:00000200:13.0:1547251672.796654:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (9)++ 00000800:00000200:13.0:1547251672.796658:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (10)-- 00000800:00000200:13.0:1547251672.796659:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (9)-- 00000800:00000200:13.0:1547251672.796661:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (8)++ 00000800:00000200:13.0:1547251672.796665:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (9)-- 00000800:00000200:13.0:1547251672.796666:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (8)-- 00000800:00000200:13.0:1547251672.796667:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (7)++ 00000800:00000200:13.0:1547251672.796671:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (8)-- 00000800:00000200:15.0:1547251672.796673:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (8)++ 00000800:00000200:13.0:1547251672.796674:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (8)-- 00000800:00000200:15.0:1547251672.796675:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (7)-- 00000800:00000200:13.0:1547251672.796676:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (7)++ 00000800:00000200:15.0:1547251672.796677:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (7)-- 00000800:00000200:15.0:1547251672.796678:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (6)++ 00000800:00000200:13.0:1547251672.796678:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (6)-- 00000800:00000200:15.0:1547251672.796679:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (6)-- 00000800:00000200:13.0:1547251672.796679:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (6)-- 00000800:00000200:15.0:1547251672.796680:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (4)-- 00000800:00000200:13.0:1547251672.796681:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (3)++ 00000800:00000200:13.0:1547251672.796685:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (4)-- 00000800:00000200:13.0:1547251672.796686:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (3)-- 00000800:00000200:13.0:1547251672.796687:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f800] (2)++ 00000800:00000200:13.0:1547251672.796690:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f800] (3)-- 00000800:00000200:13.0:1547251672.796691:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (2)-- 00000800:00000200:13.0:1547251672.796692:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f800] (1)-- 00000800:00000200:9.0:1547251672.796698:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.797571:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251672.797575:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.797592:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251672.797594:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251672.797598:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251672.797605:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251672.797848:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251672.797854:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97a99502c000 0x55502c000(0x55502c000) 00000800:00000200:10.0:1547251672.797855:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b8246a0000 0x13e46a0000(0x13e46a0000) 00000800:00000200:10.0:1547251672.797856:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ae6aa3c000 0xa2aa3c000(0xa2aa3c000) 00000800:00000200:10.0:1547251672.797856:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ae6aaa5000 0xa2aaa5000(0xa2aaa5000) 00000800:00000200:10.0:1547251672.797857:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b17b1b1000 0xd3b1b1000(0xd3b1b1000) 00000800:00000200:10.0:1547251672.797858:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b17b399000 0xd3b399000(0xd3b399000) 00000800:00000200:10.0:1547251672.797859:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97a995224000 0x555224000(0x555224000) 00000800:00000200:10.0:1547251672.797860:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ae3787e000 0x9f787e000(0x9f787e000) 00000800:00000200:10.0:1547251672.797861:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ae6ab19000 0xa2ab19000(0xa2ab19000) 00000800:00000200:10.0:1547251672.797861:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b81d76f000 0x13dd76f000(0x13dd76f000) 00000800:00000200:10.0:1547251672.797862:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ae5d2c5000 0xa1d2c5000(0xa1d2c5000) 00000800:00000200:10.0:1547251672.797863:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ae6a9fc000 0xa2a9fc000(0xa2a9fc000) 00000800:00000200:10.0:1547251672.797866:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97a65d0ea000 0x21d0ea000(0x21d0ea000) 00000800:00000200:10.0:1547251672.797867:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ae3782d000 0x9f782d000(0x9f782d000) 00000800:00000200:10.0:1547251672.797868:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97a9952e3000 0x5552e3000(0x5552e3000) 00000800:00000200:10.0:1547251672.797869:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b81d517000 0x13dd517000(0x13dd517000) 00000800:00000200:10.0:1547251672.797869:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b17b0b9000 0xd3b0b9000(0xd3b0b9000) 00000800:00000200:10.0:1547251672.797870:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b81d52c000 0x13dd52c000(0x13dd52c000) 00000800:00000200:9.0:1547251672.798431:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798432:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798433:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798434:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798434:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798435:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798436:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798436:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798437:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798437:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798438:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798439:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798439:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798440:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798441:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798441:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798442:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798443:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798443:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798444:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798444:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798445:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798446:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798446:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798447:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798447:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798448:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798449:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798449:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798450:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798451:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798452:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798452:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798453:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:9.0:1547251672.798453:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:9.0:1547251672.798454:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000100:12.0:1547251672.798718:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251672.798721:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251672.798723:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251672.798758:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:12.0:1547251672.798777:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:13.0:1547251672.798779:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (19)++ 00000800:00000200:13.0:1547251672.798785:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (20)-- 00000800:00000200:13.0:1547251672.798786:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (19)-- 00000800:00000200:13.0:1547251672.798788:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (18)-- 00000800:00000200:8.2:1547251672.798905:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07d800] (17)++ 00000800:00000200:13.0:1547251672.798919:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (18)++ 00000800:00000200:13.0:1547251672.798923:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (19)-- 00000800:00000200:13.0:1547251672.798924:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (18)-- 00000800:00000200:13.0:1547251672.798924:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (17)++ 00000800:00000200:13.0:1547251672.798925:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (18)-- 00000800:00000200:13.0:1547251672.798926:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (17)-- 00000800:00000200:13.0:1547251672.798927:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (16)++ 00000800:00000200:13.0:1547251672.798927:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (17)-- 00000800:00000200:13.0:1547251672.798928:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (16)-- 00000800:00000200:13.0:1547251672.798929:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (15)++ 00000800:00000200:13.0:1547251672.798930:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (16)-- 00000800:00000200:13.0:1547251672.798931:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (15)-- 00000800:00000200:15.0:1547251672.798932:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (14)++ 00000800:00000200:15.0:1547251672.798939:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (15)-- 00000800:00000200:15.0:1547251672.798940:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (14)-- 00000800:00000200:15.0:1547251672.798941:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (13)++ 00000800:00000200:15.0:1547251672.798951:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (14)-- 00000800:00000200:15.0:1547251672.798952:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (13)-- 00000800:00000200:15.0:1547251672.798953:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (12)++ 00000800:00000200:15.0:1547251672.798954:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (13)-- 00000800:00000200:13.0:1547251672.798955:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (13)++ 00000800:00000200:15.0:1547251672.798956:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (13)-- 00000800:00000200:15.0:1547251672.798957:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (12)++ 00000800:00000200:13.0:1547251672.798957:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (12)-- 00000800:00000200:13.0:1547251672.798958:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (12)-- 00000800:00000200:15.0:1547251672.798959:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (11)-- 00000800:00000200:13.0:1547251672.798959:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (11)++ 00000800:00000200:15.0:1547251672.798960:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (11)-- 00000800:00000200:15.0:1547251672.798961:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (10)++ 00000800:00000200:13.0:1547251672.798961:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (10)-- 00000800:00000200:13.0:1547251672.798962:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (10)-- 00000800:00000200:15.0:1547251672.798963:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (9)-- 00000800:00000200:13.0:1547251672.798963:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (9)++ 00000800:00000200:15.0:1547251672.798964:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (9)-- 00000800:00000200:15.0:1547251672.798965:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (8)++ 00000800:00000200:13.0:1547251672.798965:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (8)-- 00000800:00000200:13.0:1547251672.798966:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (8)-- 00000800:00000200:15.0:1547251672.798967:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (7)-- 00000800:00000200:13.0:1547251672.798967:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (7)++ 00000800:00000200:15.0:1547251672.798969:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (7)-- 00000800:00000200:15.0:1547251672.798970:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (6)++ 00000800:00000200:13.0:1547251672.798970:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (6)-- 00000800:00000200:13.0:1547251672.798971:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (6)-- 00000800:00000200:15.0:1547251672.798972:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (5)-- 00000800:00000200:13.0:1547251672.798972:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (5)++ 00000800:00000200:15.0:1547251672.798973:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (5)-- 00000800:00000200:15.0:1547251672.798974:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d800] (4)++ 00000800:00000200:13.0:1547251672.798974:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (4)-- 00000800:00000200:13.0:1547251672.798975:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (4)-- 00000800:00000200:15.0:1547251672.798976:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d800] (3)-- 00000800:00000200:13.0:1547251672.798976:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (3)-- 00000800:00000200:15.0:1547251672.798977:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d800] (1)-- 00000800:00000200:9.0:1547251672.798989:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.799818:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251672.799822:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.799832:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251672.799833:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251672.799839:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251672.799845:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251672.800093:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251672.800098:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba9b621000 0x165b621000(0x165b621000) 00000800:00000200:10.0:1547251672.800099:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b62c426000 0x11ec426000(0x11ec426000) 00000800:00000200:10.0:1547251672.800100:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b5872c8000 0x11472c8000(0x11472c8000) 00000800:00000200:10.0:1547251672.800101:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97badc80b000 0x169c80b000(0x169c80b000) 00000800:00000200:10.0:1547251672.800102:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ad4c2d6000 0x90c2d6000(0x90c2d6000) 00000800:00000200:10.0:1547251672.800103:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba997d1000 0x16597d1000(0x16597d1000) 00000800:00000200:10.0:1547251672.800104:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ba9ef48000 0x165ef48000(0x165ef48000) 00000800:00000200:10.0:1547251672.800104:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97baa13a2000 0x16613a2000(0x16613a2000) 00000800:00000200:10.0:1547251672.800105:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ba9d3c1000 0x165d3c1000(0x165d3c1000) 00000800:00000200:10.0:1547251672.800106:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba996e3000 0x16596e3000(0x16596e3000) 00000800:00000200:10.0:1547251672.800107:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97bad6ae0000 0x1696ae0000(0x1696ae0000) 00000800:00000200:10.0:1547251672.800108:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97a589ecb000 0x149ecb000(0x149ecb000) 00000800:00000200:10.0:1547251672.800109:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97baa0f30000 0x1660f30000(0x1660f30000) 00000800:00000200:10.0:1547251672.800110:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97baddebc000 0x169debc000(0x169debc000) 00000800:00000200:10.0:1547251672.800110:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba9ba4b000 0x165ba4b000(0x165ba4b000) 00000800:00000200:10.0:1547251672.800111:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba961ae000 0x16561ae000(0x16561ae000) 00000800:00000200:10.0:1547251672.800112:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b80cd1b000 0x13ccd1b000(0x13ccd1b000) 00000800:00000200:10.0:1547251672.800113:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97baa1dbf000 0x1661dbf000(0x1661dbf000) 00000800:00000200:9.0:1547251672.800662:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800664:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800665:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800665:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800666:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800666:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800667:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800668:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800668:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800669:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800669:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800670:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800671:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800671:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800672:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800673:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800673:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800674:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800674:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800675:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800676:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800676:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800677:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800677:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800678:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800679:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800680:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800680:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800681:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800681:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800682:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800683:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800683:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800684:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251672.800684:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251672.800685:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000100:12.0:1547251672.801014:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251672.801017:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251672.801018:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:12.0:1547251672.801063:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07f200] (19)-- 00000800:00000200:8.2:1547251672.801072:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f200] (18)++ 00000800:00000200:13.0:1547251672.801092:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:13.0:1547251672.801098:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:13.0:1547251672.801099:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (19)-- 00000800:00000200:13.0:1547251672.801101:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (18)-- 00000800:00000200:8.2:1547251672.801210:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f200] (17)++ 00000800:00000200:13.0:1547251672.801221:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (18)++ 00000800:00000200:13.0:1547251672.801226:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (19)-- 00000800:00000200:13.0:1547251672.801226:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (18)-- 00000800:00000200:13.0:1547251672.801227:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (17)++ 00000800:00000200:13.0:1547251672.801228:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (18)-- 00000800:00000200:13.0:1547251672.801229:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (17)-- 00000800:00000200:13.0:1547251672.801229:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (16)++ 00000800:00000200:13.0:1547251672.801230:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (17)-- 00000800:00000200:13.0:1547251672.801231:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (16)-- 00000800:00000200:13.0:1547251672.801232:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (15)++ 00000800:00000200:13.0:1547251672.801233:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (16)-- 00000800:00000200:13.0:1547251672.801233:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (15)-- 00000800:00000200:15.0:1547251672.801235:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (14)++ 00000800:00000200:15.0:1547251672.801242:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (15)-- 00000800:00000200:13.0:1547251672.801244:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (15)++ 00000800:00000200:15.0:1547251672.801245:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (15)-- 00000800:00000200:15.0:1547251672.801246:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (14)++ 00000800:00000200:13.0:1547251672.801246:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (14)-- 00000800:00000200:13.0:1547251672.801247:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (14)-- 00000800:00000200:15.0:1547251672.801248:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (13)-- 00000800:00000200:13.0:1547251672.801248:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (13)++ 00000800:00000200:15.0:1547251672.801249:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (13)-- 00000800:00000200:15.0:1547251672.801250:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (12)++ 00000800:00000200:13.0:1547251672.801250:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (12)-- 00000800:00000200:13.0:1547251672.801251:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (12)-- 00000800:00000200:15.0:1547251672.801252:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (11)-- 00000800:00000200:13.0:1547251672.801252:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (11)++ 00000800:00000200:15.0:1547251672.801253:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (11)-- 00000800:00000200:15.0:1547251672.801254:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (10)++ 00000800:00000200:13.0:1547251672.801254:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (10)-- 00000800:00000200:13.0:1547251672.801255:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (10)-- 00000800:00000200:15.0:1547251672.801256:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (9)-- 00000800:00000200:13.0:1547251672.801257:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (9)++ 00000800:00000200:15.0:1547251672.801258:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (9)-- 00000800:00000200:13.0:1547251672.801258:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (8)-- 00000800:00000200:15.0:1547251672.801259:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (8)++ 00000800:00000200:13.0:1547251672.801259:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (8)-- 00000800:00000200:15.0:1547251672.801260:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (7)-- 00000800:00000200:15.0:1547251672.801261:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (7)-- 00000800:00000200:13.0:1547251672.801261:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (7)++ 00000800:00000200:13.0:1547251672.801262:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (6)-- 00000800:00000200:15.0:1547251672.801263:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (6)++ 00000800:00000200:13.0:1547251672.801263:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (6)-- 00000800:00000200:15.0:1547251672.801265:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (5)-- 00000800:00000200:13.0:1547251672.801266:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (5)++ 00000800:00000200:15.0:1547251672.801267:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (5)-- 00000800:00000200:13.0:1547251672.801267:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (4)-- 00000800:00000200:15.0:1547251672.801268:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (4)++ 00000800:00000200:13.0:1547251672.801268:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (4)-- 00000800:00000200:15.0:1547251672.801270:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (3)-- 00000800:00000200:13.0:1547251672.801270:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (3)-- 00000800:00000200:15.0:1547251672.801271:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (1)-- 00000800:00000200:9.0:1547251672.801283:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.801912:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251672.801916:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.801924:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251672.801926:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251672.801931:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251672.801936:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251672.802180:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251672.802187:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba9bf33000 0x165bf33000(0x165bf33000) 00000800:00000200:10.0:1547251672.802189:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97baa0e2c000 0x1660e2c000(0x1660e2c000) 00000800:00000200:10.0:1547251672.802190:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97a72db0b000 0x2edb0b000(0x2edb0b000) 00000800:00000200:10.0:1547251672.802190:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ba8f7b5000 0x164f7b5000(0x164f7b5000) 00000800:00000200:10.0:1547251672.802191:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ae6aa9f000 0xa2aa9f000(0xa2aa9f000) 00000800:00000200:10.0:1547251672.802192:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ae6aaa0000 0xa2aaa0000(0xa2aaa0000) 00000800:00000200:10.0:1547251672.802193:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97baa134a000 0x166134a000(0x166134a000) 00000800:00000200:10.0:1547251672.802194:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97baa134b000 0x166134b000(0x166134b000) 00000800:00000200:10.0:1547251672.802195:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ba90fc4000 0x1650fc4000(0x1650fc4000) 00000800:00000200:10.0:1547251672.802195:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba90fc5000 0x1650fc5000(0x1650fc5000) 00000800:00000200:10.0:1547251672.802196:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ba941f3000 0x16541f3000(0x16541f3000) 00000800:00000200:10.0:1547251672.802197:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97bad6a8c000 0x1696a8c000(0x1696a8c000) 00000800:00000200:10.0:1547251672.802198:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba9ce49000 0x165ce49000(0x165ce49000) 00000800:00000200:10.0:1547251672.802199:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97bae1cfd000 0x16a1cfd000(0x16a1cfd000) 00000800:00000200:10.0:1547251672.802200:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba90fac000 0x1650fac000(0x1650fac000) 00000800:00000200:10.0:1547251672.802200:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97a601aa3000 0x1c1aa3000(0x1c1aa3000) 00000800:00000200:10.0:1547251672.802201:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ba95c64000 0x1655c64000(0x1655c64000) 00000800:00000200:10.0:1547251672.802202:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b17b185000 0xd3b185000(0xd3b185000) 00000800:00000200:9.0:1547251672.802757:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802758:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802759:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802760:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802760:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802761:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802762:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802762:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802763:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802764:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802764:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802765:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802765:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802766:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802767:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802767:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802768:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802768:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802769:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802770:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802770:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802771:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802772:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802773:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802773:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802774:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802774:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802775:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802776:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802776:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802777:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802777:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802778:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802779:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:9.0:1547251672.802779:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:9.0:1547251672.802780:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000100:12.0:1547251672.803082:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251672.803084:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251672.803086:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251672.803134:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:12.0:1547251672.803135:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c400] (19)-- 00000800:00000200:13.0:1547251672.803154:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (19)++ 00000800:00000200:13.0:1547251672.803161:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (20)-- 00000800:00000200:13.0:1547251672.803162:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (19)-- 00000800:00000200:13.0:1547251672.803163:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (18)-- 00000800:00000200:8.2:1547251672.803277:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c400] (17)++ 00000800:00000200:13.0:1547251672.803288:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (18)++ 00000800:00000200:13.0:1547251672.803293:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (19)-- 00000800:00000200:13.0:1547251672.803293:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (18)-- 00000800:00000200:13.0:1547251672.803294:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (17)++ 00000800:00000200:13.0:1547251672.803295:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (18)-- 00000800:00000200:13.0:1547251672.803295:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (17)-- 00000800:00000200:13.0:1547251672.803296:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (16)++ 00000800:00000200:13.0:1547251672.803297:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (17)-- 00000800:00000200:13.0:1547251672.803298:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (16)-- 00000800:00000200:13.0:1547251672.803298:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (15)++ 00000800:00000200:13.0:1547251672.803299:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (16)-- 00000800:00000200:13.0:1547251672.803300:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (15)-- 00000800:00000200:15.0:1547251672.803302:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (14)++ 00000800:00000200:15.0:1547251672.803309:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (15)-- 00000800:00000200:15.0:1547251672.803310:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (14)-- 00000800:00000200:15.0:1547251672.803311:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (13)++ 00000800:00000200:15.0:1547251672.803314:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (14)-- 00000800:00000200:15.0:1547251672.803314:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (13)-- 00000800:00000200:15.0:1547251672.803315:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (12)++ 00000800:00000200:15.0:1547251672.803316:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (13)-- 00000800:00000200:13.0:1547251672.803317:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (13)++ 00000800:00000200:15.0:1547251672.803318:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (13)-- 00000800:00000200:15.0:1547251672.803319:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (12)++ 00000800:00000200:13.0:1547251672.803319:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (12)-- 00000800:00000200:13.0:1547251672.803320:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (12)-- 00000800:00000200:15.0:1547251672.803321:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (11)-- 00000800:00000200:13.0:1547251672.803321:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (11)++ 00000800:00000200:15.0:1547251672.803322:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (11)-- 00000800:00000200:15.0:1547251672.803323:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (10)++ 00000800:00000200:13.0:1547251672.803323:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (10)-- 00000800:00000200:13.0:1547251672.803324:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (10)-- 00000800:00000200:15.0:1547251672.803325:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (9)-- 00000800:00000200:13.0:1547251672.803325:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (9)++ 00000800:00000200:15.0:1547251672.803326:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (9)-- 00000800:00000200:15.0:1547251672.803327:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (8)++ 00000800:00000200:13.0:1547251672.803327:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (8)-- 00000800:00000200:13.0:1547251672.803328:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (8)-- 00000800:00000200:15.0:1547251672.803329:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (7)-- 00000800:00000200:13.0:1547251672.803329:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (7)++ 00000800:00000200:15.0:1547251672.803330:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (7)-- 00000800:00000200:15.0:1547251672.803332:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (6)++ 00000800:00000200:13.0:1547251672.803332:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (6)-- 00000800:00000200:13.0:1547251672.803333:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (6)-- 00000800:00000200:15.0:1547251672.803334:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (5)-- 00000800:00000200:15.0:1547251672.803335:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (5)-- 00000800:00000200:13.0:1547251672.803335:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (5)++ 00000800:00000200:13.0:1547251672.803336:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (4)-- 00000800:00000200:15.0:1547251672.803337:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c400] (4)++ 00000800:00000200:13.0:1547251672.803337:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (4)-- 00000800:00000200:15.0:1547251672.803338:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c400] (3)-- 00000800:00000200:13.0:1547251672.803339:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (3)-- 00000800:00000200:15.0:1547251672.803340:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c400] (1)-- 00000800:00000200:9.0:1547251672.803351:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.804086:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251672.804090:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.804099:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251672.804100:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251672.804107:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251672.804115:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251672.804358:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251672.804364:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b17b107000 0xd3b107000(0xd3b107000) 00000800:00000200:10.0:1547251672.804366:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ae6a9f2000 0xa2a9f2000(0xa2a9f2000) 00000800:00000200:10.0:1547251672.804367:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b17b3c5000 0xd3b3c5000(0xd3b3c5000) 00000800:00000200:10.0:1547251672.804368:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ae6ab6b000 0xa2ab6b000(0xa2ab6b000) 00000800:00000200:10.0:1547251672.804368:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b82461b000 0x13e461b000(0x13e461b000) 00000800:00000200:10.0:1547251672.804369:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b81d5ad000 0x13dd5ad000(0x13dd5ad000) 00000800:00000200:10.0:1547251672.804370:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b81d787000 0x13dd787000(0x13dd787000) 00000800:00000200:10.0:1547251672.804371:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b907293000 0x14c7293000(0x14c7293000) 00000800:00000200:10.0:1547251672.804372:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ae6ab11000 0xa2ab11000(0xa2ab11000) 00000800:00000200:10.0:1547251672.804373:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b81d40b000 0x13dd40b000(0x13dd40b000) 00000800:00000200:10.0:1547251672.804374:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b81d689000 0x13dd689000(0x13dd689000) 00000800:00000200:10.0:1547251672.804375:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b81d638000 0x13dd638000(0x13dd638000) 00000800:00000200:10.0:1547251672.804376:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b9072d3000 0x14c72d3000(0x14c72d3000) 00000800:00000200:10.0:1547251672.804377:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ae6ab74000 0xa2ab74000(0xa2ab74000) 00000800:00000200:10.0:1547251672.804377:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ae6a9aa000 0xa2a9aa000(0xa2a9aa000) 00000800:00000200:10.0:1547251672.804378:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b81d767000 0x13dd767000(0x13dd767000) 00000800:00000200:10.0:1547251672.804379:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ae6a97a000 0xa2a97a000(0xa2a97a000) 00000800:00000200:10.0:1547251672.804380:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97a65d357000 0x21d357000(0x21d357000) 00000800:00000200:9.0:1547251672.804939:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804941:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804941:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804942:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804948:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804948:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804949:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804950:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804950:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804951:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804952:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804952:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804953:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804953:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804954:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804955:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804955:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804957:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804957:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804958:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804958:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804959:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804960:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804960:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804961:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804961:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804962:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804963:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804963:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804964:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804964:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804965:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804965:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804966:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251672.804967:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251672.804967:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000100:12.0:1547251672.805230:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251672.805233:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251672.805236:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251672.805273:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:13.0:1547251672.805294:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (20)++ 00000800:00000200:13.0:1547251672.805302:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (21)-- 00000800:00000200:13.0:1547251672.805303:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:13.0:1547251672.805304:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (19)-- 00000800:00000200:12.0:1547251672.805317:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:8.2:1547251672.805414:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c600] (17)++ 00000800:00000200:13.0:1547251672.805425:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)++ 00000800:00000200:13.0:1547251672.805430:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (19)-- 00000800:00000200:13.0:1547251672.805431:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:13.0:1547251672.805432:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (17)++ 00000800:00000200:13.0:1547251672.805432:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:13.0:1547251672.805433:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (17)-- 00000800:00000200:13.0:1547251672.805434:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (16)++ 00000800:00000200:13.0:1547251672.805435:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (17)-- 00000800:00000200:13.0:1547251672.805435:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (16)-- 00000800:00000200:13.0:1547251672.805436:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)++ 00000800:00000200:13.0:1547251672.805438:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (16)-- 00000800:00000200:13.0:1547251672.805438:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)-- 00000800:00000200:15.0:1547251672.805439:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (14)++ 00000800:00000200:15.0:1547251672.805447:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (15)-- 00000800:00000200:15.0:1547251672.805448:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (14)-- 00000800:00000200:15.0:1547251672.805449:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)++ 00000800:00000200:15.0:1547251672.805452:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (14)-- 00000800:00000200:15.0:1547251672.805452:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:15.0:1547251672.805453:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)++ 00000800:00000200:15.0:1547251672.805454:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:13.0:1547251672.805455:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)++ 00000800:00000200:15.0:1547251672.805456:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:15.0:1547251672.805457:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)++ 00000800:00000200:13.0:1547251672.805457:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (12)-- 00000800:00000200:13.0:1547251672.805458:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)-- 00000800:00000200:15.0:1547251672.805459:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (11)-- 00000800:00000200:13.0:1547251672.805459:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)++ 00000800:00000200:15.0:1547251672.805460:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)-- 00000800:00000200:13.0:1547251672.805460:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (10)-- 00000800:00000200:15.0:1547251672.805461:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (10)++ 00000800:00000200:13.0:1547251672.805461:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (10)-- 00000800:00000200:15.0:1547251672.805462:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (9)-- 00000800:00000200:15.0:1547251672.805463:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (9)-- 00000800:00000200:13.0:1547251672.805463:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (9)++ 00000800:00000200:13.0:1547251672.805464:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:15.0:1547251672.805465:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)++ 00000800:00000200:13.0:1547251672.805465:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:15.0:1547251672.805466:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (7)-- 00000800:00000200:13.0:1547251672.805467:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (7)++ 00000800:00000200:15.0:1547251672.805468:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (7)-- 00000800:00000200:13.0:1547251672.805468:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (6)-- 00000800:00000200:15.0:1547251672.805469:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (6)++ 00000800:00000200:13.0:1547251672.805470:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (6)-- 00000800:00000200:15.0:1547251672.805471:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (5)-- 00000800:00000200:13.0:1547251672.805471:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)++ 00000800:00000200:15.0:1547251672.805472:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)-- 00000800:00000200:15.0:1547251672.805473:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (4)++ 00000800:00000200:13.0:1547251672.805473:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (4)-- 00000800:00000200:13.0:1547251672.805474:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (4)-- 00000800:00000200:15.0:1547251672.805475:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (3)-- 00000800:00000200:13.0:1547251672.805475:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (3)-- 00000800:00000200:15.0:1547251672.805476:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (1)-- 00000800:00000200:9.0:1547251672.805488:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251672.806093:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000100:00000200:8.0:1547251673.146673:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447600, portal 4 00000100:00000200:8.0:1547251673.146676:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447600, offset 0 00000400:00000200:8.0:1547251673.146678:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251673.146681:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251673.146684:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04f70 00000400:00000200:8.0:1547251673.146686:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4cb9 [1] + 0 00000400:00000200:8.0:1547251673.146689:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ba9d367aa0 00000100:00000200:8.0:1547251673.146690:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251673.146699:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba9d367aa0 00000400:00000200:8.0:1547251673.146700:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251673.146705:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824755d00 x1622384012447600/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251680 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251673.146712:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688aa0 00000400:00000200:8.0:1547251673.146713:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251673.146715:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251673.146715:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447616, portal 4 00000100:00000200:7.0:1547251673.146719:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447616, offset 0 00000100:00000200:9.0:1547251673.146721:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447600 00000400:00000200:7.0:1547251673.146721:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251673.146724:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251673.146727:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04f80 00000400:00000200:7.0:1547251673.146730:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2d55 [1] + 0 00010000:00000200:9.0:1547251673.146731:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3d8450 x1622384012447600/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:734/0 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:7.0:1547251673.146732:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b645990198 00000100:00000200:7.0:1547251673.146734:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:9.0:1547251673.146738:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447600, offset 192 00000400:00000200:9.0:1547251673.146740:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251673.146742:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990198 00000400:00000200:9.0:1547251673.146743:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251673.146743:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251673.146745:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df7800 x1622384012447616/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251680 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251673.146746:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04f70 00000400:00000200:9.0:1547251673.146748:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf4f1 [1] + 192 00000400:00000200:7.0:1547251673.146750:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea880 00000100:00000200:9.0:1547251673.146751:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824755d00 x1622384012447600/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251680 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251673.146751:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251673.146752:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:9.0:1547251673.146759:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251673.146761:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea3b8 00000400:00000200:9.0:1547251673.146762:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251673.146764:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251673.146768:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824755d00 x1622384012447600/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251680 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251673.146771:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447616 00000100:00000200:9.0:1547251673.146772:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:8.0:1547251673.146774:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824755d00 x1622384012447600/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251680 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251673.146780:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6882a8 00010000:00000200:5.0:1547251673.146780:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11e450 x1622384012447616/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:734/0 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Interpret:/0/0 rc 0/0 00010000:00080000:8.0:1547251673.146783:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824755d00 x1622384012447600/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251680 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251673.146787:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447616, offset 192 00000400:00000200:5.0:1547251673.146790:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251673.146793:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251673.146796:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04f80 00000400:00000200:5.0:1547251673.146798:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf501 [1] + 192 00000100:00000200:5.0:1547251673.146801:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df7800 x1622384012447616/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251680 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251673.146811:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251673.146813:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad088 00000400:00000200:5.0:1547251673.146814:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251673.146815:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251673.146819:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df7800 x1622384012447616/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251680 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251673.146820:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251673.146824:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df7800 x1622384012447616/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251680 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251673.146828:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea330 00010000:00080000:7.0:1547251673.146831:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df7800 x1622384012447616/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251680 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251673.162675:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447632, portal 4 00000100:00000200:7.0:1547251673.162678:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447632, offset 0 00000400:00000200:7.0:1547251673.162680:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251673.162683:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251673.162686:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c04f90 00000400:00000200:7.0:1547251673.162688:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2d85 [1] + 0 00000400:00000200:7.0:1547251673.162690:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17eaa18 00000100:00000200:7.0:1547251673.162692:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:7.0:1547251673.162700:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eaa18 00000400:00000200:7.0:1547251673.162701:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251673.162703:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df4500 x1622384012447632/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251680 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251673.162713:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea880 00000400:00000200:7.0:1547251673.162714:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251673.162715:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251673.162733:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447632 00010000:00000200:5.0:1547251673.162742:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11e050 x1622384012447632/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:734/0 lens 224/368 e 0 to 0 dl 1547251679 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251673.162749:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447632, offset 192 00000400:00000200:5.0:1547251673.162751:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251673.162755:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251673.162757:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c04f90 00000400:00000200:5.0:1547251673.162760:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf511 [1] + 192 00000100:00000200:5.0:1547251673.162763:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df4500 x1622384012447632/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251680 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251673.162772:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251673.162774:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adf68 00000400:00000200:5.0:1547251673.162775:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251673.162776:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251673.162781:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df4500 x1622384012447632/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251680 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251673.162781:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:7.0:1547251673.162786:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df4500 x1622384012447632/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251680 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251673.162789:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea330 00010000:00080000:7.0:1547251673.162792:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df4500 x1622384012447632/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251680 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:10.0:1547251673.307687:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:10.0:1547251673.307703:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:10.0:1547251673.307705:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000400:10.0:1547251673.307707:0:12658:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.2.10.184@o2ib10: 3 seconds 00000400:00000200:10.0:1547251673.307709:0:12658:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = -110, hstatus = 6 00000400:00000200:10.0:1547251673.307712:0:12658:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.2.10.184@o2ib10: PUT: LOCAL_TIMEOUT 00000800:00000200:9.0:1547251673.307712:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000400:00000100:10.0:1547251673.307714:0:12658:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.2.10.184@o2ib10 exceeded retry count 0 00000400:00000200:10.0:1547251673.307717:0:12658:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06aa18 00000800:00000200:9.0:1547251673.307721:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:9.0:1547251673.308221:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:9.0:1547251673.308228:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ae6a97a000 0xa2a97a000(0xa2a97a000) 00000800:00000200:9.0:1547251673.308230:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b81d767000 0x13dd767000(0x13dd767000) 00000800:00000200:9.0:1547251673.308231:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ae6a9aa000 0xa2a9aa000(0xa2a9aa000) 00000800:00000200:9.0:1547251673.308232:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ae6ab74000 0xa2ab74000(0xa2ab74000) 00000800:00000200:9.0:1547251673.308233:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b9072d3000 0x14c72d3000(0x14c72d3000) 00000800:00000200:9.0:1547251673.308234:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b81d638000 0x13dd638000(0x13dd638000) 00000800:00000200:9.0:1547251673.308235:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b81d689000 0x13dd689000(0x13dd689000) 00000800:00000200:9.0:1547251673.308235:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b81d40b000 0x13dd40b000(0x13dd40b000) 00000800:00000200:9.0:1547251673.308236:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ae6ab11000 0xa2ab11000(0xa2ab11000) 00000800:00000200:9.0:1547251673.308237:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b907293000 0x14c7293000(0x14c7293000) 00000800:00000200:9.0:1547251673.308238:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b81d787000 0x13dd787000(0x13dd787000) 00000800:00000200:9.0:1547251673.308239:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b81d5ad000 0x13dd5ad000(0x13dd5ad000) 00000800:00000200:9.0:1547251673.308240:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b82461b000 0x13e461b000(0x13e461b000) 00000800:00000200:9.0:1547251673.308240:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ae6ab6b000 0xa2ab6b000(0xa2ab6b000) 00000800:00000200:9.0:1547251673.308241:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b17b3c5000 0xd3b3c5000(0xd3b3c5000) 00000800:00000200:9.0:1547251673.308242:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ae6a9f2000 0xa2a9f2000(0xa2a9f2000) 00000800:00000200:9.0:1547251673.308243:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b17b107000 0xd3b107000(0xd3b107000) 00000800:00000200:9.0:1547251673.308244:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b82459e000 0x13e459e000(0x13e459e000) 00000800:00000200:9.0:1547251673.308899:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308901:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308901:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308902:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308903:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308903:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308904:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308904:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308905:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308906:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308906:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308908:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308908:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308909:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308909:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308910:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308911:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308911:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308912:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308912:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308913:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308914:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308914:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308915:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308915:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308916:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308917:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308917:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308918:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308918:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308919:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308920:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308920:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308921:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:9.0:1547251673.308921:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:9.0:1547251673.308922:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000100:12.0:1547251673.309257:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: don't reconnect (no need), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251673.309260:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251673.309262:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:12.0:1547251673.309264:0:70:0:(o2iblnd.c:417:kiblnd_unlink_peer_locked()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (2)-- 00000400:00000200:12.0:1547251673.309268:0:70:0:(router.c:1756:lnet_notify()) 10.1.11.90@o2ib10 notifying 10.2.10.184@o2ib10: down 00000400:00000200:12.0:1547251673.309269:0:70:0:(router.c:120:lnet_notify_locked()) Old news 00000800:00000200:8.2:1547251673.309317:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:12.0:1547251673.309329:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:13.0:1547251673.309344:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (19)++ 00000800:00000200:13.0:1547251673.309352:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (20)-- 00000800:00000200:13.0:1547251673.309354:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (19)-- 00000800:00000200:13.0:1547251673.309355:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (18)-- 00000800:00000200:8.2:1547251673.309461:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba8baebe00] (17)++ 00000800:00000200:13.0:1547251673.309474:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (18)++ 00000800:00000200:13.0:1547251673.309478:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (19)-- 00000800:00000200:13.0:1547251673.309480:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (18)-- 00000800:00000200:13.0:1547251673.309481:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (17)++ 00000800:00000200:13.0:1547251673.309482:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (18)-- 00000800:00000200:13.0:1547251673.309482:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (17)-- 00000800:00000200:13.0:1547251673.309483:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (16)++ 00000800:00000200:13.0:1547251673.309484:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (17)-- 00000800:00000200:13.0:1547251673.309485:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (16)-- 00000800:00000200:15.0:1547251673.309487:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (15)++ 00000800:00000200:15.0:1547251673.309494:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (16)-- 00000800:00000200:15.0:1547251673.309495:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (15)-- 00000800:00000200:15.0:1547251673.309496:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (14)++ 00000800:00000200:15.0:1547251673.309499:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (15)-- 00000800:00000200:15.0:1547251673.309500:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (14)-- 00000800:00000200:15.0:1547251673.309500:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (13)++ 00000800:00000200:15.0:1547251673.309503:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (14)-- 00000800:00000200:13.0:1547251673.309503:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (14)++ 00000800:00000200:15.0:1547251673.309504:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (14)-- 00000800:00000200:15.0:1547251673.309505:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (13)++ 00000800:00000200:13.0:1547251673.309505:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (13)-- 00000800:00000200:13.0:1547251673.309506:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (13)-- 00000800:00000200:15.0:1547251673.309507:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (12)-- 00000800:00000200:13.0:1547251673.309507:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (12)++ 00000800:00000200:15.0:1547251673.309508:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (12)-- 00000800:00000200:15.0:1547251673.309509:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (11)++ 00000800:00000200:13.0:1547251673.309509:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (11)-- 00000800:00000200:13.0:1547251673.309510:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (11)-- 00000800:00000200:15.0:1547251673.309511:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (10)-- 00000800:00000200:13.0:1547251673.309511:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (10)++ 00000800:00000200:15.0:1547251673.309512:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (10)-- 00000800:00000200:13.0:1547251673.309513:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (9)-- 00000800:00000200:15.0:1547251673.309514:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (9)++ 00000800:00000200:13.0:1547251673.309514:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (9)-- 00000800:00000200:15.0:1547251673.309515:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (8)-- 00000800:00000200:13.0:1547251673.309516:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (8)++ 00000800:00000200:15.0:1547251673.309517:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (8)-- 00000800:00000200:13.0:1547251673.309517:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (7)-- 00000800:00000200:15.0:1547251673.309518:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (7)++ 00000800:00000200:15.0:1547251673.309519:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (6)-- 00000800:00000200:13.0:1547251673.309519:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (7)-- 00000800:00000200:15.0:1547251673.309520:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (6)-- 00000800:00000200:13.0:1547251673.309520:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (6)++ 00000800:00000200:13.0:1547251673.309521:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (5)-- 00000800:00000200:15.0:1547251673.309522:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (5)++ 00000800:00000200:13.0:1547251673.309522:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (5)-- 00000800:00000200:15.0:1547251673.309523:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (4)-- 00000800:00000200:13.0:1547251673.309523:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baebe00] (4)++ 00000800:00000200:15.0:1547251673.309524:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (4)-- 00000800:00000200:13.0:1547251673.309525:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baebe00] (3)-- 00000800:00000200:15.0:1547251673.309526:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (3)-- 00000800:00000200:13.0:1547251673.309527:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baebe00] (1)-- 00000800:00000200:9.0:1547251673.310479:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0280] -> 10.2.10.184@o2ib10 (1)-- 00000100:00000200:3.0:1547251675.178682:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447664, portal 4 00000100:00000200:10.0:1547251675.178685:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447648, portal 4 00000100:00000200:3.0:1547251675.178685:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447664, offset 0 00000100:00000200:10.0:1547251675.178688:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447648, offset 0 00000400:00000200:3.0:1547251675.178688:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:10.0:1547251675.178691:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:3.0:1547251675.178691:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:10.0:1547251675.178693:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:3.0:1547251675.178694:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23091] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:10.0:1547251675.178696:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23091] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251675.178697:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23090] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23091] 00000400:00000200:10.0:1547251675.178699:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23090] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23091] 00000400:00000200:3.0:1547251675.178699:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:10.0:1547251675.178701:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251675.178701:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251675.178702:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251675.178703:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251675.178704:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251675.178704:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9767, 9767] 00000400:00000200:10.0:1547251675.178706:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9767, 9767] 00000400:00000200:3.0:1547251675.178706:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:10.0:1547251675.178708:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:3.0:1547251675.178710:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251675.178713:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:3.0:1547251675.178716:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000400:00000200:10.0:1547251675.178717:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251675.178717:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:3.0:1547251675.178718:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:10.0:1547251675.178720:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:3.0:1547251675.178721:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:10.0:1547251675.178722:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:10.0:1547251675.178724:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:10.0:1547251675.178726:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:10.0:1547251675.178729:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:9.2:1547251675.178741:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:7.2:1547251675.178750:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:4.0:1547251675.178753:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447680, portal 4 00000100:00000200:4.0:1547251675.178757:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447680, offset 0 00000400:00000200:4.0:1547251675.178761:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:4.0:1547251675.178764:0:12674:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:4.0:1547251675.178767:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23092] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:4.0:1547251675.178770:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23091] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23092] 00000400:00000200:4.0:1547251675.178772:0:12674:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:4.0:1547251675.178773:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251675.178775:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251675.178776:0:12674:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9768, 9768] 00000400:00000200:4.0:1547251675.178778:0:12674:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:4.0:1547251675.178783:0:12674:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:4.0:1547251675.178785:0:12674:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:4.0:1547251675.178789:0:12674:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:4.0:1547251675.178790:0:12674:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:4.0:1547251675.178791:0:12674:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:4.0:1547251675.178794:0:12674:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (24)-- 00000800:00000200:15.0:1547251675.178796:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:15.0:1547251675.178799:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (24)-- 00000100:00000200:15.0:1547251675.178803:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c2400 x1622384012447664/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251682 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251675.178808:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245555d8 00000400:00000200:15.0:1547251675.178809:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:13.0:1547251675.178811:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000400:00000200:15.0:1547251675.178812:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251675.178813:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:15.0:1547251675.178814:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:15.0:1547251675.178815:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:15.0:1547251675.178816:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:13.0:1547251675.178817:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960eb300 x1622384012447648/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251682 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:15.0:1547251675.178818:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37935d00 x1622384012447680/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251682 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251675.178822:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a9146e8 00000400:00000200:15.0:1547251675.178823:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b440 00000400:00000200:15.0:1547251675.178823:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251675.178824:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:15.0:1547251675.178826:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:13.0:1547251675.178826:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:15.0:1547251675.178827:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:15.0:1547251675.178828:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:13.0:1547251675.178829:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:15.0:1547251675.178830:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:7.2:1547251675.178891:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:9.2:1547251675.178896:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:13.0:1547251675.178903:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:13.0:1547251675.178906:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000800:00000200:15.0:1547251675.178909:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:13.0:1547251675.178910:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:15.0:1547251675.178912:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.93@o2ib10 00000800:00000200:7.2:1547251675.178913:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000400:00000200:13.0:1547251675.178915:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04fa0 00000400:00000200:15.0:1547251675.178916:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251675.178918:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7ffed [1] + 192 00000400:00000200:15.0:1547251675.178919:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04fb0 00000100:00000200:13.0:1547251675.178921:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960eb300 x1622384012447648/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251682 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251675.178932:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:15.0:1547251675.178934:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7ffdd [1] + 192 00000800:00000200:13.0:1547251675.178934:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:13.0:1547251675.178936:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:13.0:1547251675.178936:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:15.0:1547251675.178937:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c2400 x1622384012447664/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251682 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251675.178938:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:13.0:1547251675.178940:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:13.0:1547251675.178941:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:13.0:1547251675.178950:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:13.0:1547251675.178953:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251675.178955:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:13.0:1547251675.178956:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04fc0 00000800:00000200:15.0:1547251675.178957:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (21)++ 00000100:00000200:10.0:1547251675.178958:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960eb300 x1622384012447648/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251682 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251675.178959:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:15.0:1547251675.178960:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:15.0:1547251675.178961:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000100:00000200:10.0:1547251675.178963:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960eb300 x1622384012447648/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251682 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251675.178967:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a9145d8 00000400:00000200:13.0:1547251675.178970:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x7fffd [1] + 192 00010000:00080000:10.0:1547251675.178971:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba960eb300 x1622384012447648/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251682 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251675.178973:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c2400 x1622384012447664/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251682 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251675.178979:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c2400 x1622384012447664/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251682 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251675.178983:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00000100:00000200:13.0:1547251675.178985:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37935d00 x1622384012447680/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251682 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:3.0:1547251675.178987:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c2400 x1622384012447664/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251682 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:13.0:1547251675.178993:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251675.178995:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (21)++ 00000800:00000200:13.0:1547251675.178996:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (22)-- 00000800:00000200:13.0:1547251675.178997:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:4.0:1547251675.179004:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37935d00 x1622384012447680/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251682 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251675.179009:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37935d00 x1622384012447680/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251682 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251675.179014:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bc38 00010000:00080000:4.0:1547251675.179017:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ae37935d00 x1622384012447680/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251682 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251677.290679:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447696, portal 4 00000100:00000200:3.0:1547251677.290682:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447696, offset 0 00000400:00000200:3.0:1547251677.290686:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:3.0:1547251677.290689:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:3.0:1547251677.290692:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23092] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251677.290695:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23092] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23092] 00000400:00000200:3.0:1547251677.290697:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251677.290699:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251677.290700:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251677.290701:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9773, 9773] 00000400:00000200:3.0:1547251677.290703:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:3.0:1547251677.290708:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251677.290710:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:3.0:1547251677.290713:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:3.0:1547251677.290715:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:3.0:1547251677.290716:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:3.0:1547251677.290718:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000100:00000200:12.0:1547251677.290739:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447712, portal 4 00000100:00000200:9.0:1547251677.290741:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447728, portal 4 00000100:00000200:12.0:1547251677.290744:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447712, offset 0 00000100:00000200:9.0:1547251677.290744:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447728, offset 0 00000400:00000200:9.0:1547251677.290747:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:12.0:1547251677.290748:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251677.290750:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:12.0:1547251677.290751:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251677.290752:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23093] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251677.290754:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23093] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251677.290755:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23092] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23093] 00000400:00000200:12.0:1547251677.290757:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23092] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23093] 00000400:00000200:9.0:1547251677.290757:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251677.290759:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251677.290760:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251677.290760:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251677.290761:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251677.290762:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251677.290763:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9773, 9774] 00000400:00000200:12.0:1547251677.290764:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9773, 9774] 00000400:00000200:9.0:1547251677.290765:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:12.0:1547251677.290766:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:9.0:1547251677.290769:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:5.2:1547251677.290771:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:9.0:1547251677.290772:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:12.0:1547251677.290775:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251677.290778:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:9.0:1547251677.290779:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251677.290781:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251677.290781:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:9.0:1547251677.290782:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:12.0:1547251677.290783:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (21)++ 00000800:00000200:12.0:1547251677.290785:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (23)++ 00000800:00000200:9.0:1547251677.290785:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (23)-- 00000800:00000200:12.0:1547251677.290788:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251677.290794:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251677.290803:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000100:00000200:11.0:1547251677.290809:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c2100 x1622384012447696/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251677.290810:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:11.0:1547251677.290816:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555bb0 00000400:00000200:11.0:1547251677.290818:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251677.290822:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251677.290825:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:7.2:1547251677.290835:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251677.290850:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:10.0:1547251677.290857:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000100:00000200:10.0:1547251677.290862:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf4200 x1622384012447728/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251677.290863:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:11.0:1547251677.290864:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000100:00000200:11.0:1547251677.290868:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a6300 x1622384012447712/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251677.290868:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea990 00000400:00000200:10.0:1547251677.290869:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251677.290873:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251677.290875:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038220 00000800:00000200:10.0:1547251677.290875:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000400:00000200:11.0:1547251677.290876:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251677.290877:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:11.0:1547251677.290879:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251677.290880:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:5.2:1547251677.290904:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251677.290910:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:5.2:1547251677.290916:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:10.0:1547251677.290920:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251677.290922:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251677.290923:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251677.290925:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251677.290928:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251677.290929:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251677.290933:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04ff0 00000400:00000200:10.0:1547251677.290933:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04fd0 00000400:00000200:10.0:1547251677.290936:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd75f9 [1] + 192 00000400:00000200:11.0:1547251677.290941:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7619 [1] + 192 00000100:00000200:10.0:1547251677.290952:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c2100 x1622384012447696/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251677.290962:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:11.0:1547251677.290964:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf4200 x1622384012447728/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251677.290964:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251677.290966:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:10.0:1547251677.290968:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251677.290970:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251677.290972:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251677.290974:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251677.290976:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000100:00000200:3.0:1547251677.290977:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c2100 x1622384012447696/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251677.290978:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251677.290979:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:10.0:1547251677.290980:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251677.290981:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:3.0:1547251677.290982:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c2100 x1622384012447696/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251677.290983:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000400:00000200:10.0:1547251677.290983:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c04fe0 00000800:00000200:11.0:1547251677.290984:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:3.0:1547251677.290986:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00000100:00000200:9.0:1547251677.290989:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf4200 x1622384012447728/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:3.0:1547251677.290990:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c2100 x1622384012447696/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251677.290995:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf4200 x1622384012447728/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251677.290999:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaee0 00000400:00000200:10.0:1547251677.291002:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7609 [1] + 192 00010000:00080000:9.0:1547251677.291002:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf4200 x1622384012447728/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251677.291006:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a6300 x1622384012447712/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251677.291016:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251677.291017:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251677.291019:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251677.291019:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:12.0:1547251677.291027:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a6300 x1622384012447712/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251677.291032:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a6300 x1622384012447712/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251677.291037:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038dd0 00010000:00080000:12.0:1547251677.291041:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a6300 x1622384012447712/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251677.754681:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447744, portal 4 00000100:00000200:9.0:1547251677.754684:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447744, offset 0 00000400:00000200:9.0:1547251677.754687:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251677.754690:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251677.754692:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23093] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251677.754695:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23094] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23093] 00000400:00000200:9.0:1547251677.754698:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251677.754699:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251677.754700:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251677.754702:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9766, 9767] 00000400:00000200:9.0:1547251677.754703:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000100:00000200:2.0:1547251677.754704:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447776, portal 4 00000100:00000200:2.0:1547251677.754707:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447776, offset 0 00000400:00000200:9.0:1547251677.754708:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:2.0:1547251677.754710:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:9.0:1547251677.754711:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:2.0:1547251677.754713:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:9.0:1547251677.754714:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251677.754715:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000400:00000200:2.0:1547251677.754715:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23094] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:9.0:1547251677.754717:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000400:00000200:2.0:1547251677.754718:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23094] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23094] 00000800:00000200:9.0:1547251677.754720:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000400:00000200:2.0:1547251677.754722:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251677.754723:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251677.754725:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251677.754726:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9767, 9767] 00000800:00000200:12.2:1547251677.754728:0:12682:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000400:00000200:2.0:1547251677.754728:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:2.0:1547251677.754732:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251677.754735:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:2.0:1547251677.754738:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251677.754740:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:2.0:1547251677.754741:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:2.0:1547251677.754743:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000100:00000200:12.0:1547251677.754749:0:12682:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447760, portal 4 00000100:00000200:12.0:1547251677.754751:0:12682:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447760, offset 0 00000400:00000200:12.0:1547251677.754755:0:12682:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:12.0:1547251677.754757:0:12682:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:0.2:1547251677.754758:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000400:00000200:12.0:1547251677.754760:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23094] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251677.754763:0:12682:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23095] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23094] 00000400:00000200:12.0:1547251677.754765:0:12682:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:12.0:1547251677.754767:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251677.754768:0:12682:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251677.754769:0:12682:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9767, 9768] 00000400:00000200:12.0:1547251677.754771:0:12682:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:12.0:1547251677.754776:0:12682:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251677.754779:0:12682:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:12.0:1547251677.754782:0:12682:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251677.754783:0:12682:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:12.0:1547251677.754784:0:12682:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (23)++ 00000800:00000200:4.0:1547251677.754786:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (24)++ 00000800:00000200:12.0:1547251677.754787:0:12682:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (24)-- 00000800:00000200:5.0:1547251677.754790:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251677.754790:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (24)-- 00000800:00000200:5.0:1547251677.754792:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000100:00000200:4.0:1547251677.754794:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf5d00 x1622384012447744/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251677.754796:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f3c00 x1622384012447776/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251677.754803:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06aa18 00000400:00000200:5.0:1547251677.754805:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251677.754808:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251677.754808:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea770 00000400:00000200:4.0:1547251677.754809:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:5.0:1547251677.754811:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:4.0:1547251677.754812:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251677.754813:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (23)++ 00000800:00000200:5.0:1547251677.754815:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (24)-- 00000800:00000200:4.0:1547251677.754815:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (24)-- 00000800:00000200:4.0:1547251677.754816:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000100:00000200:5.0:1547251677.754818:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a3900 x1622384012447760/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251677.754818:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:5.0:1547251677.754823:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038bb0 00000400:00000200:5.0:1547251677.754825:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251677.754828:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251677.754829:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:12.2:1547251677.754901:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:0.2:1547251677.754907:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251677.754913:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:4.0:1547251677.754916:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251677.754920:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000800:00000200:5.0:1547251677.754921:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251677.754923:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251677.754925:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251677.754928:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251677.754929:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05020 00000400:00000200:5.0:1547251677.754932:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05000 00000400:00000200:4.0:1547251677.754934:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4545 [1] + 192 00000400:00000200:5.0:1547251677.754939:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4535 [1] + 192 00000100:00000200:4.0:1547251677.754942:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f3c00 x1622384012447776/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251677.754964:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251677.754966:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf5d00 x1622384012447744/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251677.754966:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251677.754968:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251677.754970:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251677.754972:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251677.754974:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251677.754977:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251677.754978:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:5.0:1547251677.754979:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000100:00000200:2.0:1547251677.754979:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f3c00 x1622384012447776/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251677.754981:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251677.754982:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:4.0:1547251677.754982:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05010 00000800:00000200:5.0:1547251677.754984:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:5.0:1547251677.754985:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:2.0:1547251677.754985:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f3c00 x1622384012447776/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251677.754987:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:2.0:1547251677.754989:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06aaa0 00000400:00000200:4.0:1547251677.754992:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4555 [1] + 192 00010000:00080000:2.0:1547251677.754992:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f3c00 x1622384012447776/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251677.754997:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf5d00 x1622384012447744/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251677.755002:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf5d00 x1622384012447744/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251677.755007:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaee0 00000100:00000200:4.0:1547251677.755009:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a3900 x1622384012447760/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251677.755010:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf5d00 x1622384012447744/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:4.0:1547251677.755019:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251677.755020:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251677.755022:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251677.755023:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000100:00000200:12.0:1547251677.755028:0:12682:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a3900 x1622384012447760/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251677.755034:0:12682:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a3900 x1622384012447760/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251677.755038:0:12682:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038dd0 00010000:00080000:12.0:1547251677.755042:0:12682:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a3900 x1622384012447760/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251684 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251678.154674:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447792, portal 4 00000100:00000200:9.0:1547251678.154678:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447792, offset 0 00000400:00000200:9.0:1547251678.154680:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251678.154683:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251678.154686:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c05030 00000400:00000200:9.0:1547251678.154688:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4d99 [1] + 0 00000400:00000200:9.0:1547251678.154690:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d6884c8 00000100:00000200:9.0:1547251678.154692:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251678.154700:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6884c8 00000400:00000200:9.0:1547251678.154701:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251678.154705:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf6900 x1622384012447792/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251685 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251678.154711:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea088 00000400:00000200:9.0:1547251678.154712:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251678.154713:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251678.154723:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447792 00010000:00000200:8.0:1547251678.154733:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c3d8050 x1622384012447792/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:739/0 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251678.154736:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447808, portal 4 00000100:00000200:5.0:1547251678.154740:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447808, offset 0 00000100:00000200:8.0:1547251678.154741:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447792, offset 192 00000400:00000200:8.0:1547251678.154743:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251678.154744:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251678.154746:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251678.154748:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c05030 00000400:00000200:5.0:1547251678.154748:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251678.154751:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf551 [1] + 192 00000400:00000200:5.0:1547251678.154751:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c05040 00000100:00000200:8.0:1547251678.154753:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf6900 x1622384012447792/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251685 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251678.154753:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2df5 [1] + 0 00000400:00000200:5.0:1547251678.154756:0:12675:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17eabb0 00000100:00000200:5.0:1547251678.154757:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251678.154761:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251678.154764:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6882a8 00000400:00000200:8.0:1547251678.154765:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251678.154767:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:5.0:1547251678.154767:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eabb0 00000400:00000200:5.0:1547251678.154768:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251678.154770:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37937200 x1622384012447808/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251685 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251678.154771:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf6900 x1622384012447792/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251685 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251678.154774:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:5.0:1547251678.154775:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ade58 00000400:00000200:5.0:1547251678.154776:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:9.0:1547251678.154777:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf6900 x1622384012447792/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251685 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251678.154777:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:9.0:1547251678.154782:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaee0 00010000:00080000:9.0:1547251678.154786:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf6900 x1622384012447792/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251685 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251678.154797:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447808 00010000:00000200:6.0:1547251678.154806:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11dc50 x1622384012447808/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:739/0 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251678.154813:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447808, offset 192 00000400:00000200:6.0:1547251678.154815:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251678.154818:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251678.154821:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c05040 00000400:00000200:6.0:1547251678.154823:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf561 [1] + 192 00000100:00000200:6.0:1547251678.154827:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37937200 x1622384012447808/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251685 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251678.154836:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251678.154838:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990660 00000400:00000200:6.0:1547251678.154839:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251678.154840:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251678.154844:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37937200 x1622384012447808/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251685 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251678.154846:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251678.154849:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37937200 x1622384012447808/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251685 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251678.154853:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad330 00010000:00080000:5.0:1547251678.154856:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37937200 x1622384012447808/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251685 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251678.170680:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447824, portal 4 00000100:00000200:6.0:1547251678.170683:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447824, offset 0 00000400:00000200:6.0:1547251678.170685:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251678.170688:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251678.170691:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c05050 00000400:00000200:6.0:1547251678.170693:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2e25 [1] + 0 00000400:00000200:6.0:1547251678.170695:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97bae17ead48 00000100:00000200:6.0:1547251678.170697:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251678.170707:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ead48 00000400:00000200:6.0:1547251678.170708:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251678.170710:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37930f00 x1622384012447824/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251685 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251678.170715:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990d48 00000400:00000200:6.0:1547251678.170715:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251678.170717:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251678.170730:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447824 00010000:00000200:4.0:1547251678.170742:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11d850 x1622384012447824/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:739/0 lens 224/368 e 0 to 0 dl 1547251684 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251678.170758:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012447824, offset 192 00000400:00000200:4.0:1547251678.170760:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251678.170763:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251678.170766:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c05050 00000400:00000200:4.0:1547251678.170769:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf571 [1] + 192 00000100:00000200:4.0:1547251678.170772:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37930f00 x1622384012447824/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251685 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251678.170782:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251678.170784:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040bc38 00000400:00000200:4.0:1547251678.170785:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251678.170786:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251678.170790:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37930f00 x1622384012447824/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251685 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251678.170791:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251678.170795:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37930f00 x1622384012447824/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251685 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251678.170799:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990c38 00010000:00080000:6.0:1547251678.170802:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37930f00 x1622384012447824/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251685 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:0.2:1547251679.642328:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b81d647a00] (20)++ 00000800:00000200:15.0:1547251679.642383:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:15.0:1547251679.642394:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.10.184@o2ib10 00000400:00000200:15.0:1547251679.642398:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.10.184@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251679.642402:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.10.184@o2ib10 of length 520 into portal 26 MB=0x5c3934d100070 00000400:00000200:15.0:1547251679.642405:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.10.184@o2ib10 of length 520/520 into md 0xd5629 [1] + 0 00000400:00000200:15.0:1547251679.642407:0:12666:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52dd48 00000100:00000200:15.0:1547251679.642410:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:15.0:1547251679.642415:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52dd48 00000400:00000200:15.0:1547251679.642416:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251679.642418:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b81d647a00] (22)++ 00000800:00000200:15.0:1547251679.642419:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b81d647a00] (23)-- 00000800:00000200:15.0:1547251679.642420:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)-- 00000800:00000200:15.0:1547251679.642422:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (21)-- 00000100:00000200:2.0:1547251679.642474:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622412304056432 00010000:02000400:2.0:1547251679.642481:0:12929:0:(ldlm_lib.c:1152:target_handle_connect()) MGS: Received new LWP connection from 10.0.10.184@o2ib10, removing former export from same NID 00010000:00080000:2.0:1547251679.642486:0:12929:0:(ldlm_lib.c:1228:target_handle_connect()) MGS: connection from 8534aa65-7ac8-198c-0718-3e6f83d57705@10.0.10.184@o2ib10 t0 exp ffff97b9b3928400 cur 1547251679 last 1547251679 00000400:00000200:2.0:1547251679.642493:0:12929:0:(peer.c:1106:LNetPrimaryNID()) NID 10.0.10.184@o2ib10 primary NID 10.0.10.184@o2ib10 rc 0 00000100:00080000:2.0:1547251679.642495:0:12929:0:(import.c:104:ptlrpc_import_enter_resend()) ffff97b824577000 : changing import state from FULL to RECOVER 00000100:00080000:2.0:1547251679.642497:0:12929:0:(import.c:1528:ptlrpc_import_recovery_state_machine()) ffff97b824577000 : changing import state from RECOVER to FULL 00000100:02000000:2.0:1547251679.642499:0:12929:0:(import.c:1534:ptlrpc_import_recovery_state_machine()) MGS: Connection restored to f7adbd22-dd00-137c-616b-9c8e12a50a01 (at 10.0.10.184@o2ib10) 00010000:00000200:2.0:1547251679.642503:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692feb850 x1622412304056432/t0(0) o250->8534aa65-7ac8-198c-0718-3e6f83d57705@10.0.10.184@o2ib10:0/0 lens 520/416 e 0 to 0 dl 1547251779 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:2.0:1547251679.642509:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 25, xid 1622412304056432, offset 0 00000400:00000200:2.0:1547251679.642512:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.10.184@o2ib10 00000400:00000200:2.0:1547251679.642515:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.0.10.184@o2ib10 local destination 00000400:00000200:2.0:1547251679.642518:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251679.642519:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251679.642520:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.0.10.184@o2ib10 c:[8, 8], s:[19, 20] 00000400:00000200:2.0:1547251679.642522:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.10.184@o2ib10 00000400:00000200:2.0:1547251679.642528:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.10.184@o2ib10(10.0.10.184@o2ib10:10.0.10.184@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251679.642531:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 384 bytes in 1 frags to 12345-10.0.10.184@o2ib10 00000800:00000200:2.0:1547251679.642534:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b971916c00] -> 10.0.10.184@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251679.642535:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b81d647a00] (20)++ 00000800:00000200:2.0:1547251679.642536:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:2.0:1547251679.642539:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b81d647a00] (22)-- 00000100:00000200:2.0:1547251679.642543:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:0.2:1547251679.642551:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b81d647a00] (21)++ 00000800:00000200:15.0:1547251679.642564:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)++ 00000800:00000200:15.0:1547251679.642569:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b81d647a00] (23)-- 00000400:00000200:15.0:1547251679.642572:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06aaa0 00000400:00000200:15.0:1547251679.642573:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:15.0:1547251679.642575:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.10.184@o2ib10: PUT: OK 00000800:00000200:13.0:1547251679.642576:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (22)-- 00000800:00000200:15.0:1547251679.642579:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b81d647a00] (21)-- 00000800:00000200:8.2:1547251679.646968:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d02b200] (20)++ 00000800:00000200:13.0:1547251679.647021:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d02b200] (21)++ 00000800:00000200:13.0:1547251679.647032:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.10.184@o2ib10 00000400:00000200:13.0:1547251679.647035:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.10.184@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251679.647039:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.10.184@o2ib10 of length 224 into portal 26 MB=0x5c3934d100080 00000400:00000200:13.0:1547251679.647043:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.10.184@o2ib10 of length 224/224 into md 0xd5639 [1] + 0 00000400:00000200:13.0:1547251679.647045:0:12665:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52d770 00000100:00000200:13.0:1547251679.647047:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:13.0:1547251679.647052:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52d770 00000400:00000200:13.0:1547251679.647053:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251679.647056:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d02b200] (22)++ 00000800:00000200:13.0:1547251679.647057:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d02b200] (23)-- 00000800:00000200:13.0:1547251679.647058:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b200] (22)-- 00000800:00000200:13.0:1547251679.647060:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d02b200] (21)-- 00000100:00000200:2.0:1547251679.647108:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622412304056448 00000020:00000080:2.0:1547251679.647116:0:12929:0:(genops.c:1503:class_disconnect()) disconnect: cookie 0xdec335f6b872ab88 00000100:00080000:2.0:1547251679.647119:0:12929:0:(import.c:1739:ptlrpc_cleanup_imp()) ffff97b824577000 : changing import state from FULL to CLOSED 00010000:00000200:2.0:1547251679.647126:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692fea450 x1622412304056448/t0(0) o251->8534aa65-7ac8-198c-0718-3e6f83d57705@10.0.10.184@o2ib10:740/0 lens 224/224 e 0 to 0 dl 1547251685 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:2.0:1547251679.647133:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622412304056448, offset 192 00000400:00000200:2.0:1547251679.647136:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.10.184@o2ib10 00000400:00000200:2.0:1547251679.647140:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.10.184@o2ib10 local destination 00000400:00000200:2.0:1547251679.647142:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251679.647143:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251679.647145:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.0.10.184@o2ib10 c:[8, 8], s:[20, 20] 00000400:00000200:2.0:1547251679.647146:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.2.10.184@o2ib10 00000400:00000200:2.0:1547251679.647151:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.2.10.184@o2ib10(10.0.10.184@o2ib10:10.2.10.184@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251679.647154:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.2.10.184@o2ib10 00000800:00000200:2.0:1547251679.647158:0:12929:0:(o2iblnd_cb.c:1604:kiblnd_launch_tx()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (1)++ 00000800:00000200:2.0:1547251679.647161:0:12929:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:2.0:1547251679.647173:0:12929:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:2.0:1547251679.647175:0:12929:0:(o2iblnd_cb.c:1611:kiblnd_launch_tx()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000100:00000200:2.0:1547251679.647179:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:9.0:1547251679.647192:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:9.0:1547251679.647201:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000020:00000080:6.0:1547251679.647205:0:12925:0:(genops.c:1024:class_export_put()) final put ffff97b9b3928400/8534aa65-7ac8-198c-0718-3e6f83d57705 00000020:00000080:6.0:1547251679.647211:0:40:0:(genops.c:974:class_export_destroy()) destroying export ffff97b9b3928400/8534aa65-7ac8-198c-0718-3e6f83d57705 for MGS 00000020:00000080:6.0:1547251679.647219:0:40:0:(genops.c:1224:obd_zombie_import_free()) destroying import ffff97b824577000 for MGS 00000800:00000200:9.0:1547251679.647510:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:9.0:1547251679.647517:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b17b3c5000 0xd3b3c5000(0xd3b3c5000) 00000800:00000200:9.0:1547251679.647519:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ae6ab6b000 0xa2ab6b000(0xa2ab6b000) 00000800:00000200:9.0:1547251679.647520:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b82461b000 0x13e461b000(0x13e461b000) 00000800:00000200:9.0:1547251679.647520:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b81d5ad000 0x13dd5ad000(0x13dd5ad000) 00000800:00000200:9.0:1547251679.647521:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b81d787000 0x13dd787000(0x13dd787000) 00000800:00000200:9.0:1547251679.647522:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b907293000 0x14c7293000(0x14c7293000) 00000800:00000200:9.0:1547251679.647523:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ae6ab11000 0xa2ab11000(0xa2ab11000) 00000800:00000200:9.0:1547251679.647524:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b81d40b000 0x13dd40b000(0x13dd40b000) 00000800:00000200:9.0:1547251679.647525:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b81d689000 0x13dd689000(0x13dd689000) 00000800:00000200:9.0:1547251679.647526:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b81d638000 0x13dd638000(0x13dd638000) 00000800:00000200:9.0:1547251679.647527:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b9072d3000 0x14c72d3000(0x14c72d3000) 00000800:00000200:9.0:1547251679.647527:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ae6ab74000 0xa2ab74000(0xa2ab74000) 00000800:00000200:9.0:1547251679.647528:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ae6a9aa000 0xa2a9aa000(0xa2a9aa000) 00000800:00000200:9.0:1547251679.647529:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b81d767000 0x13dd767000(0x13dd767000) 00000800:00000200:9.0:1547251679.647530:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ae6a97a000 0xa2a97a000(0xa2a97a000) 00000800:00000200:9.0:1547251679.647531:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba9eec4000 0x165eec4000(0x165eec4000) 00000800:00000200:9.0:1547251679.647532:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97bae1d25000 0x16a1d25000(0x16a1d25000) 00000800:00000200:9.0:1547251679.647533:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97bae1d24000 0x16a1d24000(0x16a1d24000) 00000800:00000200:9.0:1547251679.648162:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648163:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648164:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648165:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648165:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648166:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648166:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648167:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648168:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648168:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648169:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648169:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648170:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648171:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648172:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648173:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648173:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648174:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648174:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648175:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648176:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648176:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648177:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648177:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648178:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648179:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648179:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648180:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648180:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648181:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648182:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648182:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648183:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648184:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:9.0:1547251679.648184:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:9.0:1547251679.648185:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000100:12.0:1547251679.648432:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251679.648435:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251679.648436:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251679.648475:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba8bae8e00] (19)++ 00000800:00000200:12.0:1547251679.648498:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:13.0:1547251679.648499:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (20)++ 00000800:00000200:13.0:1547251679.648508:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (20)-- 00000800:00000200:13.0:1547251679.648509:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (19)-- 00000800:00000200:13.0:1547251679.648511:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (18)-- 00000800:00000200:8.2:1547251679.648562:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba8bae8e00] (17)++ 00000800:00000200:13.0:1547251679.648574:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (18)++ 00000800:00000200:13.0:1547251679.648579:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (19)-- 00000800:00000200:13.0:1547251679.648580:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (18)-- 00000800:00000200:13.0:1547251679.648581:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (17)++ 00000800:00000200:13.0:1547251679.648582:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (18)-- 00000800:00000200:13.0:1547251679.648582:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (17)-- 00000800:00000200:13.0:1547251679.648604:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (16)++ 00000800:00000200:13.0:1547251679.648610:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (17)-- 00000800:00000200:13.0:1547251679.648611:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (16)-- 00000800:00000200:13.0:1547251679.648613:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (15)++ 00000800:00000200:13.0:1547251679.648614:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (16)-- 00000800:00000200:13.0:1547251679.648615:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (15)-- 00000800:00000200:13.0:1547251679.648616:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (14)++ 00000800:00000200:13.0:1547251679.648621:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (15)-- 00000800:00000200:13.0:1547251679.648622:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (14)-- 00000800:00000200:13.0:1547251679.648623:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (13)++ 00000800:00000200:13.0:1547251679.648627:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (14)-- 00000800:00000200:13.0:1547251679.648628:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (13)-- 00000800:00000200:13.0:1547251679.648630:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (12)++ 00000800:00000200:13.0:1547251679.648634:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (13)-- 00000800:00000200:13.0:1547251679.648635:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (12)-- 00000800:00000200:13.0:1547251679.648636:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (11)++ 00000800:00000200:13.0:1547251679.648640:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (12)-- 00000800:00000200:13.0:1547251679.648641:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (11)-- 00000800:00000200:13.0:1547251679.648642:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (10)++ 00000800:00000200:13.0:1547251679.648646:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (11)-- 00000800:00000200:13.0:1547251679.648648:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (10)-- 00000800:00000200:13.0:1547251679.648649:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (9)++ 00000800:00000200:13.0:1547251679.648653:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (10)-- 00000800:00000200:13.0:1547251679.648655:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (9)-- 00000800:00000200:15.0:1547251679.648657:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (8)++ 00000800:00000200:15.0:1547251679.648664:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (9)-- 00000800:00000200:15.0:1547251679.648666:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (8)-- 00000800:00000200:15.0:1547251679.648667:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (7)++ 00000800:00000200:15.0:1547251679.648670:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (8)-- 00000800:00000200:15.0:1547251679.648671:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (7)-- 00000800:00000200:15.0:1547251679.648671:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (6)++ 00000800:00000200:15.0:1547251679.648673:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (7)-- 00000800:00000200:15.0:1547251679.648673:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (6)-- 00000800:00000200:13.0:1547251679.648674:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (5)++ 00000800:00000200:13.0:1547251679.648678:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (6)-- 00000800:00000200:13.0:1547251679.648680:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (5)-- 00000800:00000200:13.0:1547251679.648681:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (4)++ 00000800:00000200:13.0:1547251679.648685:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (5)-- 00000800:00000200:13.0:1547251679.648686:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (4)-- 00000800:00000200:13.0:1547251679.648687:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (3)++ 00000800:00000200:13.0:1547251679.648689:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (4)-- 00000800:00000200:13.0:1547251679.648690:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (3)-- 00000800:00000200:13.0:1547251679.648691:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (2)++ 00000800:00000200:13.0:1547251679.648692:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8bae8e00] (3)-- 00000800:00000200:15.0:1547251679.648693:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (3)-- 00000800:00000200:13.0:1547251679.648694:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8bae8e00] (1)-- 00000800:00000200:14.0:1547251679.648720:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.649596:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251679.649603:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.649616:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251679.649618:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:2.0:1547251679.649627:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:2.0:1547251679.649636:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:2.0:1547251679.649870:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:2.0:1547251679.649880:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b1a46a7000 0xd646a7000(0xd646a7000) 00000800:00000200:2.0:1547251679.649881:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97a65d341000 0x21d341000(0x21d341000) 00000800:00000200:2.0:1547251679.649882:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ae6a970000 0xa2a970000(0xa2a970000) 00000800:00000200:2.0:1547251679.649883:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ad572f9000 0x9172f9000(0x9172f9000) 00000800:00000200:2.0:1547251679.649884:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97a65d29d000 0x21d29d000(0x21d29d000) 00000800:00000200:2.0:1547251679.649885:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b82463f000 0x13e463f000(0x13e463f000) 00000800:00000200:2.0:1547251679.649886:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ae6a83a000 0xa2a83a000(0xa2a83a000) 00000800:00000200:2.0:1547251679.649886:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b81d4e9000 0x13dd4e9000(0x13dd4e9000) 00000800:00000200:2.0:1547251679.649887:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b8247c0000 0x13e47c0000(0x13e47c0000) 00000800:00000200:2.0:1547251679.649888:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba92fef000 0x1652fef000(0x1652fef000) 00000800:00000200:2.0:1547251679.649889:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97a65d22a000 0x21d22a000(0x21d22a000) 00000800:00000200:2.0:1547251679.649890:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b630fa6000 0x11f0fa6000(0x11f0fa6000) 00000800:00000200:2.0:1547251679.649891:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba955fc000 0x16555fc000(0x16555fc000) 00000800:00000200:2.0:1547251679.649891:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba955fd000 0x16555fd000(0x16555fd000) 00000800:00000200:2.0:1547251679.649892:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b7f4704000 0x13b4704000(0x13b4704000) 00000800:00000200:2.0:1547251679.649893:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b7f4705000 0x13b4705000(0x13b4705000) 00000800:00000200:2.0:1547251679.649894:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b17b1a0000 0xd3b1a0000(0xd3b1a0000) 00000800:00000200:2.0:1547251679.649895:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b17b1a1000 0xd3b1a1000(0xd3b1a1000) 00000800:00000200:9.0:1547251679.650564:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650566:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650567:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650568:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650568:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650569:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650569:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650570:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650571:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650571:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650572:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650573:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650574:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650574:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650575:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650575:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650576:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650577:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650577:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650578:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650578:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650579:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650580:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650580:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650581:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650582:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650582:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650583:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650599:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650600:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650601:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650602:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650603:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650604:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:9.0:1547251679.650605:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b82583b000] (19)++ 00000800:00000200:9.0:1547251679.650606:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000100:12.0:1547251679.650914:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251679.650917:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251679.650918:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251679.650953:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b82583b000] (19)++ 00000800:00000200:12.0:1547251679.650978:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97b82583b000] (20)-- 00000800:00000200:15.0:1547251679.651009:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (19)++ 00000800:00000200:15.0:1547251679.651018:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (20)-- 00000800:00000200:15.0:1547251679.651019:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (19)-- 00000800:00000200:15.0:1547251679.651020:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (18)-- 00000800:00000200:8.2:1547251679.651109:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b82583b000] (17)++ 00000800:00000200:15.0:1547251679.651120:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (18)++ 00000800:00000200:15.0:1547251679.651125:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (19)-- 00000800:00000200:15.0:1547251679.651126:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (18)-- 00000800:00000200:15.0:1547251679.651126:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (17)++ 00000800:00000200:15.0:1547251679.651127:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (18)-- 00000800:00000200:15.0:1547251679.651128:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (17)-- 00000800:00000200:15.0:1547251679.651129:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (16)++ 00000800:00000200:15.0:1547251679.651129:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (17)-- 00000800:00000200:15.0:1547251679.651130:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (16)-- 00000800:00000200:15.0:1547251679.651131:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (15)++ 00000800:00000200:15.0:1547251679.651132:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (16)-- 00000800:00000200:15.0:1547251679.651133:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (15)-- 00000800:00000200:13.0:1547251679.651134:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (14)++ 00000800:00000200:13.0:1547251679.651140:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (15)-- 00000800:00000200:13.0:1547251679.651140:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (14)-- 00000800:00000200:13.0:1547251679.651142:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (13)++ 00000800:00000200:13.0:1547251679.651145:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (14)-- 00000800:00000200:13.0:1547251679.651145:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (13)-- 00000800:00000200:13.0:1547251679.651146:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (12)++ 00000800:00000200:13.0:1547251679.651147:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (13)-- 00000800:00000200:15.0:1547251679.651148:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (13)++ 00000800:00000200:13.0:1547251679.651148:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (13)-- 00000800:00000200:15.0:1547251679.651149:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (12)-- 00000800:00000200:13.0:1547251679.651150:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (12)++ 00000800:00000200:15.0:1547251679.651151:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (12)-- 00000800:00000200:15.0:1547251679.651152:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (11)++ 00000800:00000200:13.0:1547251679.651152:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (11)-- 00000800:00000200:13.0:1547251679.651153:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (11)-- 00000800:00000200:15.0:1547251679.651154:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (10)-- 00000800:00000200:13.0:1547251679.651154:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (10)++ 00000800:00000200:15.0:1547251679.651155:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (10)-- 00000800:00000200:15.0:1547251679.651156:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (9)++ 00000800:00000200:13.0:1547251679.651156:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (9)-- 00000800:00000200:13.0:1547251679.651157:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (9)-- 00000800:00000200:15.0:1547251679.651158:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (8)-- 00000800:00000200:13.0:1547251679.651158:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (8)++ 00000800:00000200:15.0:1547251679.651159:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (8)-- 00000800:00000200:15.0:1547251679.651160:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (7)++ 00000800:00000200:13.0:1547251679.651160:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (7)-- 00000800:00000200:13.0:1547251679.651161:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (7)-- 00000800:00000200:15.0:1547251679.651162:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (6)-- 00000800:00000200:13.0:1547251679.651162:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (6)++ 00000800:00000200:15.0:1547251679.651163:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (6)-- 00000800:00000200:15.0:1547251679.651164:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (5)++ 00000800:00000200:13.0:1547251679.651164:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (5)-- 00000800:00000200:13.0:1547251679.651166:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (5)-- 00000800:00000200:15.0:1547251679.651167:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (4)-- 00000800:00000200:13.0:1547251679.651167:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b82583b000] (4)++ 00000800:00000200:15.0:1547251679.651168:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (4)-- 00000800:00000200:15.0:1547251679.651169:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (3)-- 00000800:00000200:13.0:1547251679.651169:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97b82583b000] (3)-- 00000800:00000200:13.0:1547251679.651170:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b82583b000] (1)-- 00000800:00000200:9.0:1547251679.651183:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.651941:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251679.651951:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.651962:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251679.651963:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251679.651971:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251679.651979:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251679.652170:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251679.652177:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97badb94f000 0x169b94f000(0x169b94f000) 00000800:00000200:10.0:1547251679.652178:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b158afd000 0xd18afd000(0xd18afd000) 00000800:00000200:10.0:1547251679.652179:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97badc907000 0x169c907000(0x169c907000) 00000800:00000200:10.0:1547251679.652180:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b073a9d000 0xc33a9d000(0xc33a9d000) 00000800:00000200:10.0:1547251679.652181:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b6fde3e000 0x12bde3e000(0x12bde3e000) 00000800:00000200:10.0:1547251679.652182:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba92fc4000 0x1652fc4000(0x1652fc4000) 00000800:00000200:10.0:1547251679.652183:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97badc9ea000 0x169c9ea000(0x169c9ea000) 00000800:00000200:10.0:1547251679.652183:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b9073ab000 0x14c73ab000(0x14c73ab000) 00000800:00000200:10.0:1547251679.652184:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ba95590000 0x1655590000(0x1655590000) 00000800:00000200:10.0:1547251679.652185:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b17b3bf000 0xd3b3bf000(0xd3b3bf000) 00000800:00000200:10.0:1547251679.652186:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b17b1a7000 0xd3b1a7000(0xd3b1a7000) 00000800:00000200:10.0:1547251679.652187:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b17b2fe000 0xd3b2fe000(0xd3b2fe000) 00000800:00000200:10.0:1547251679.652188:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b824611000 0x13e4611000(0x13e4611000) 00000800:00000200:10.0:1547251679.652189:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b81d63e000 0x13dd63e000(0x13dd63e000) 00000800:00000200:10.0:1547251679.652189:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97a99520e000 0x55520e000(0x55520e000) 00000800:00000200:10.0:1547251679.652190:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b81d497000 0x13dd497000(0x13dd497000) 00000800:00000200:10.0:1547251679.652191:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b81d52c000 0x13dd52c000(0x13dd52c000) 00000800:00000200:10.0:1547251679.652192:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b17b0b9000 0xd3b0b9000(0xd3b0b9000) 00000800:00000200:9.0:1547251679.652740:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652741:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652742:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652742:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652743:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652744:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652745:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652745:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652746:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652747:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652747:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652748:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652749:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652749:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652750:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652750:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652751:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652752:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652752:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652753:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652753:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652754:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652755:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652755:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652756:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652756:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652757:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652758:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652758:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652759:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652759:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652760:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652761:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652761:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:9.0:1547251679.652762:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:9.0:1547251679.652763:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000100:12.0:1547251679.653066:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251679.653070:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251679.653071:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251679.653106:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:12.0:1547251679.653121:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:15.0:1547251679.653158:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (19)++ 00000800:00000200:15.0:1547251679.653167:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (20)-- 00000800:00000200:15.0:1547251679.653168:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (19)-- 00000800:00000200:15.0:1547251679.653169:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (18)-- 00000800:00000200:8.2:1547251679.653263:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07ca00] (17)++ 00000800:00000200:15.0:1547251679.653276:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (18)++ 00000800:00000200:15.0:1547251679.653280:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (19)-- 00000800:00000200:15.0:1547251679.653281:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (18)-- 00000800:00000200:15.0:1547251679.653282:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (17)++ 00000800:00000200:15.0:1547251679.653282:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (18)-- 00000800:00000200:15.0:1547251679.653283:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (17)-- 00000800:00000200:15.0:1547251679.653284:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (16)++ 00000800:00000200:15.0:1547251679.653285:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (17)-- 00000800:00000200:15.0:1547251679.653285:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (16)-- 00000800:00000200:15.0:1547251679.653286:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (15)++ 00000800:00000200:15.0:1547251679.653289:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (16)-- 00000800:00000200:15.0:1547251679.653290:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (15)-- 00000800:00000200:15.0:1547251679.653291:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (14)++ 00000800:00000200:15.0:1547251679.653292:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (15)-- 00000800:00000200:15.0:1547251679.653293:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (14)-- 00000800:00000200:13.0:1547251679.653294:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (13)++ 00000800:00000200:13.0:1547251679.653299:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (14)-- 00000800:00000200:13.0:1547251679.653300:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (13)-- 00000800:00000200:13.0:1547251679.653301:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (12)++ 00000800:00000200:13.0:1547251679.653304:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (13)-- 00000800:00000200:13.0:1547251679.653305:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (12)-- 00000800:00000200:13.0:1547251679.653306:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (11)++ 00000800:00000200:13.0:1547251679.653307:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (12)-- 00000800:00000200:15.0:1547251679.653308:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (12)++ 00000800:00000200:13.0:1547251679.653308:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (12)-- 00000800:00000200:15.0:1547251679.653309:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (11)-- 00000800:00000200:15.0:1547251679.653310:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (11)-- 00000800:00000200:13.0:1547251679.653310:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (11)++ 00000800:00000200:13.0:1547251679.653311:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (10)-- 00000800:00000200:15.0:1547251679.653312:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (10)++ 00000800:00000200:15.0:1547251679.653313:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (9)-- 00000800:00000200:13.0:1547251679.653313:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (10)-- 00000800:00000200:13.0:1547251679.653314:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (9)++ 00000800:00000200:15.0:1547251679.653315:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (9)-- 00000800:00000200:15.0:1547251679.653316:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (8)++ 00000800:00000200:13.0:1547251679.653316:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (8)-- 00000800:00000200:13.0:1547251679.653317:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (8)-- 00000800:00000200:15.0:1547251679.653318:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (7)-- 00000800:00000200:13.0:1547251679.653318:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (7)++ 00000800:00000200:15.0:1547251679.653319:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (7)-- 00000800:00000200:13.0:1547251679.653319:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (6)-- 00000800:00000200:15.0:1547251679.653320:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (6)++ 00000800:00000200:13.0:1547251679.653320:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (6)-- 00000800:00000200:15.0:1547251679.653321:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (5)-- 00000800:00000200:13.0:1547251679.653321:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (5)++ 00000800:00000200:15.0:1547251679.653322:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (5)-- 00000800:00000200:15.0:1547251679.653323:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07ca00] (4)++ 00000800:00000200:13.0:1547251679.653323:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (4)-- 00000800:00000200:13.0:1547251679.653324:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (4)-- 00000800:00000200:15.0:1547251679.653325:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07ca00] (3)-- 00000800:00000200:13.0:1547251679.653326:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (3)-- 00000800:00000200:15.0:1547251679.653327:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07ca00] (1)-- 00000800:00000200:9.0:1547251679.653339:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.654012:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251679.654017:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.654025:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251679.654027:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251679.654033:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251679.654040:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251679.654548:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251679.654557:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ae3782d000 0x9f782d000(0x9f782d000) 00000800:00000200:10.0:1547251679.654558:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97a65d0ea000 0x21d0ea000(0x21d0ea000) 00000800:00000200:10.0:1547251679.654559:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ae6a9fc000 0xa2a9fc000(0xa2a9fc000) 00000800:00000200:10.0:1547251679.654560:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ae5d2c5000 0xa1d2c5000(0xa1d2c5000) 00000800:00000200:10.0:1547251679.654561:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b81d76f000 0x13dd76f000(0x13dd76f000) 00000800:00000200:10.0:1547251679.654561:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ae6ab19000 0xa2ab19000(0xa2ab19000) 00000800:00000200:10.0:1547251679.654562:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ae3787e000 0x9f787e000(0x9f787e000) 00000800:00000200:10.0:1547251679.654563:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97a995224000 0x555224000(0x555224000) 00000800:00000200:10.0:1547251679.654564:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b17b399000 0xd3b399000(0xd3b399000) 00000800:00000200:10.0:1547251679.654565:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b17b1b1000 0xd3b1b1000(0xd3b1b1000) 00000800:00000200:10.0:1547251679.654566:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ba9cef0000 0x165cef0000(0x165cef0000) 00000800:00000200:10.0:1547251679.654566:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba9ef48000 0x165ef48000(0x165ef48000) 00000800:00000200:10.0:1547251679.654567:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97baa13a2000 0x16613a2000(0x16613a2000) 00000800:00000200:10.0:1547251679.654568:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba9d3c1000 0x165d3c1000(0x165d3c1000) 00000800:00000200:10.0:1547251679.654569:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba996e3000 0x16596e3000(0x16596e3000) 00000800:00000200:10.0:1547251679.654570:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97bad6ae0000 0x1696ae0000(0x1696ae0000) 00000800:00000200:10.0:1547251679.654571:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97a589ecb000 0x149ecb000(0x149ecb000) 00000800:00000200:10.0:1547251679.654572:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97baa0f30000 0x1660f30000(0x1660f30000) 00000800:00000200:9.0:1547251679.655138:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655139:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655140:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655141:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655142:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655142:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655143:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655144:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655144:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655145:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655145:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655146:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655147:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655147:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655148:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655149:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655149:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655150:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655150:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655151:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655151:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655152:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655153:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655153:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655154:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655154:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655155:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655156:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655156:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655157:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655157:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655158:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655159:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655159:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:9.0:1547251679.655160:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:9.0:1547251679.655160:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000100:12.0:1547251679.655479:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251679.655483:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251679.655484:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:12.0:1547251679.655537:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07e800] (19)-- 00000800:00000200:8.2:1547251679.655537:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:13.0:1547251679.655562:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (19)++ 00000800:00000200:13.0:1547251679.655570:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (20)-- 00000800:00000200:13.0:1547251679.655571:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (19)-- 00000800:00000200:13.0:1547251679.655573:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (18)-- 00000800:00000200:8.2:1547251679.655678:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e800] (17)++ 00000800:00000200:13.0:1547251679.655691:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (18)++ 00000800:00000200:13.0:1547251679.655696:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (19)-- 00000800:00000200:13.0:1547251679.655697:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (18)-- 00000800:00000200:13.0:1547251679.655697:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (17)++ 00000800:00000200:13.0:1547251679.655698:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (18)-- 00000800:00000200:13.0:1547251679.655699:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (17)-- 00000800:00000200:13.0:1547251679.655699:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (16)++ 00000800:00000200:13.0:1547251679.655700:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (17)-- 00000800:00000200:13.0:1547251679.655701:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (16)-- 00000800:00000200:13.0:1547251679.655702:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (15)++ 00000800:00000200:13.0:1547251679.655703:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (16)-- 00000800:00000200:15.0:1547251679.655705:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (16)++ 00000800:00000200:13.0:1547251679.655707:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (16)-- 00000800:00000200:15.0:1547251679.655708:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (15)-- 00000800:00000200:13.0:1547251679.655708:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (15)++ 00000800:00000200:15.0:1547251679.655709:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (15)-- 00000800:00000200:15.0:1547251679.655710:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (14)++ 00000800:00000200:13.0:1547251679.655710:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (14)-- 00000800:00000200:13.0:1547251679.655711:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (14)-- 00000800:00000200:15.0:1547251679.655712:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (13)-- 00000800:00000200:13.0:1547251679.655713:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (13)++ 00000800:00000200:15.0:1547251679.655714:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (13)-- 00000800:00000200:15.0:1547251679.655715:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (12)++ 00000800:00000200:13.0:1547251679.655715:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (12)-- 00000800:00000200:13.0:1547251679.655716:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (12)-- 00000800:00000200:15.0:1547251679.655717:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (11)-- 00000800:00000200:13.0:1547251679.655717:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (11)++ 00000800:00000200:15.0:1547251679.655718:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (11)-- 00000800:00000200:13.0:1547251679.655718:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (10)-- 00000800:00000200:15.0:1547251679.655719:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (10)++ 00000800:00000200:15.0:1547251679.655720:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (9)-- 00000800:00000200:13.0:1547251679.655720:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (10)-- 00000800:00000200:13.0:1547251679.655721:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (9)++ 00000800:00000200:15.0:1547251679.655722:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (9)-- 00000800:00000200:13.0:1547251679.655722:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (8)-- 00000800:00000200:15.0:1547251679.655723:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (8)++ 00000800:00000200:13.0:1547251679.655723:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (8)-- 00000800:00000200:15.0:1547251679.655724:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (7)-- 00000800:00000200:15.0:1547251679.655725:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (7)-- 00000800:00000200:13.0:1547251679.655725:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (7)++ 00000800:00000200:13.0:1547251679.655726:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (6)-- 00000800:00000200:15.0:1547251679.655727:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (6)++ 00000800:00000200:13.0:1547251679.655727:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (6)-- 00000800:00000200:15.0:1547251679.655728:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (5)-- 00000800:00000200:13.0:1547251679.655728:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (5)++ 00000800:00000200:15.0:1547251679.655730:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (5)-- 00000800:00000200:13.0:1547251679.655730:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (4)-- 00000800:00000200:15.0:1547251679.655731:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e800] (4)++ 00000800:00000200:15.0:1547251679.655732:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e800] (3)-- 00000800:00000200:13.0:1547251679.655732:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (4)-- 00000800:00000200:13.0:1547251679.655733:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (3)-- 00000800:00000200:15.0:1547251679.655734:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e800] (1)-- 00000800:00000200:9.0:1547251679.655748:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.656493:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251679.656497:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.656506:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251679.656507:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251679.656513:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251679.656520:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251679.656815:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251679.656820:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba9ba4b000 0x165ba4b000(0x165ba4b000) 00000800:00000200:10.0:1547251679.656821:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba961ae000 0x16561ae000(0x16561ae000) 00000800:00000200:10.0:1547251679.656822:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b80cd1b000 0x13ccd1b000(0x13ccd1b000) 00000800:00000200:10.0:1547251679.656823:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97baa1dbf000 0x1661dbf000(0x1661dbf000) 00000800:00000200:10.0:1547251679.656824:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97badd53d000 0x169d53d000(0x169d53d000) 00000800:00000200:10.0:1547251679.656825:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ae6aaa5000 0xa2aaa5000(0xa2aaa5000) 00000800:00000200:10.0:1547251679.656826:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ae6aa3c000 0xa2aa3c000(0xa2aa3c000) 00000800:00000200:10.0:1547251679.656826:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b8246a0000 0x13e46a0000(0x13e46a0000) 00000800:00000200:10.0:1547251679.656828:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97a99502c000 0x55502c000(0x55502c000) 00000800:00000200:10.0:1547251679.656829:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba90fc2000 0x1650fc2000(0x1650fc2000) 00000800:00000200:10.0:1547251679.656830:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ba90fc3000 0x1650fc3000(0x1650fc3000) 00000800:00000200:10.0:1547251679.656831:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba9727a000 0x165727a000(0x165727a000) 00000800:00000200:10.0:1547251679.656832:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba9727b000 0x165727b000(0x165727b000) 00000800:00000200:10.0:1547251679.656833:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b1233de000 0xce33de000(0xce33de000) 00000800:00000200:10.0:1547251679.656834:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b1233df000 0xce33df000(0xce33df000) 00000800:00000200:10.0:1547251679.656835:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba8cbba000 0x164cbba000(0x164cbba000) 00000800:00000200:10.0:1547251679.656835:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ba8cbbb000 0x164cbbb000(0x164cbbb000) 00000800:00000200:10.0:1547251679.656836:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97badf3ac000 0x169f3ac000(0x169f3ac000) 00000800:00000200:9.0:1547251679.657376:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657378:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657378:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657379:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657380:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657380:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657381:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657381:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657382:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657383:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657383:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657384:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657385:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657385:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657386:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657386:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657387:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657388:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657388:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657389:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657389:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657390:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657391:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657391:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657392:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657392:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657393:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657394:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657394:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657395:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657395:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657396:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657396:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657397:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:9.0:1547251679.657398:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:9.0:1547251679.657399:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000100:12.0:1547251679.657629:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251679.657632:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251679.657633:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:12.0:1547251679.657679:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07e600] (19)-- 00000800:00000200:8.2:1547251679.657689:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e600] (18)++ 00000800:00000200:13.0:1547251679.657709:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (19)++ 00000800:00000200:13.0:1547251679.657715:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (20)-- 00000800:00000200:13.0:1547251679.657716:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (19)-- 00000800:00000200:13.0:1547251679.657718:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (18)-- 00000800:00000200:8.2:1547251679.657822:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e600] (17)++ 00000800:00000200:13.0:1547251679.657833:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (18)++ 00000800:00000200:13.0:1547251679.657838:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (19)-- 00000800:00000200:13.0:1547251679.657838:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (18)-- 00000800:00000200:13.0:1547251679.657839:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (17)++ 00000800:00000200:13.0:1547251679.657840:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (18)-- 00000800:00000200:13.0:1547251679.657840:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (17)-- 00000800:00000200:13.0:1547251679.657841:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (16)++ 00000800:00000200:13.0:1547251679.657842:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (17)-- 00000800:00000200:13.0:1547251679.657842:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (16)-- 00000800:00000200:13.0:1547251679.657843:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (15)++ 00000800:00000200:13.0:1547251679.657844:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (16)-- 00000800:00000200:13.0:1547251679.657845:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (15)-- 00000800:00000200:15.0:1547251679.657846:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (14)++ 00000800:00000200:15.0:1547251679.657854:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (15)-- 00000800:00000200:15.0:1547251679.657855:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (14)-- 00000800:00000200:15.0:1547251679.657856:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (13)++ 00000800:00000200:15.0:1547251679.657858:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (14)-- 00000800:00000200:15.0:1547251679.657859:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (13)-- 00000800:00000200:15.0:1547251679.657860:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (12)++ 00000800:00000200:15.0:1547251679.657861:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (13)-- 00000800:00000200:13.0:1547251679.657861:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (13)++ 00000800:00000200:15.0:1547251679.657863:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (13)-- 00000800:00000200:15.0:1547251679.657864:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (12)++ 00000800:00000200:13.0:1547251679.657864:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (12)-- 00000800:00000200:13.0:1547251679.657865:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (12)-- 00000800:00000200:15.0:1547251679.657866:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (11)-- 00000800:00000200:13.0:1547251679.657866:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (11)++ 00000800:00000200:15.0:1547251679.657867:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (11)-- 00000800:00000200:15.0:1547251679.657868:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (10)++ 00000800:00000200:13.0:1547251679.657868:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (10)-- 00000800:00000200:13.0:1547251679.657869:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (10)-- 00000800:00000200:15.0:1547251679.657870:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (9)-- 00000800:00000200:13.0:1547251679.657871:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (9)++ 00000800:00000200:15.0:1547251679.657872:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (9)-- 00000800:00000200:13.0:1547251679.657872:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (8)-- 00000800:00000200:15.0:1547251679.657873:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (8)++ 00000800:00000200:13.0:1547251679.657875:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (8)-- 00000800:00000200:15.0:1547251679.657876:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (7)-- 00000800:00000200:13.0:1547251679.657876:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (7)++ 00000800:00000200:15.0:1547251679.657877:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (7)-- 00000800:00000200:15.0:1547251679.657878:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (6)++ 00000800:00000200:13.0:1547251679.657878:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (6)-- 00000800:00000200:13.0:1547251679.657879:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (6)-- 00000800:00000200:15.0:1547251679.657880:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (5)-- 00000800:00000200:13.0:1547251679.657880:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (5)++ 00000800:00000200:15.0:1547251679.657881:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (5)-- 00000800:00000200:15.0:1547251679.657882:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e600] (4)++ 00000800:00000200:13.0:1547251679.657882:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (4)-- 00000800:00000200:13.0:1547251679.657883:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (4)-- 00000800:00000200:15.0:1547251679.657884:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e600] (3)-- 00000800:00000200:13.0:1547251679.657884:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (3)-- 00000800:00000200:15.0:1547251679.657885:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e600] (1)-- 00000800:00000200:9.0:1547251679.657897:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.658620:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251679.658625:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.658634:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251679.658636:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251679.658644:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251679.658650:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251679.658986:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251679.658995:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b8246a7000 0x13e46a7000(0x13e46a7000) 00000800:00000200:10.0:1547251679.658997:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97ba986ec000 0x16586ec000(0x16586ec000) 00000800:00000200:10.0:1547251679.658998:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba9eec4000 0x165eec4000(0x165eec4000) 00000800:00000200:10.0:1547251679.658998:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ba986ed000 0x16586ed000(0x16586ed000) 00000800:00000200:10.0:1547251679.658999:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b5bdd33000 0x117dd33000(0x117dd33000) 00000800:00000200:10.0:1547251679.659000:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ae6a97a000 0xa2a97a000(0xa2a97a000) 00000800:00000200:10.0:1547251679.659001:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b81d767000 0x13dd767000(0x13dd767000) 00000800:00000200:10.0:1547251679.659002:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ae6a9aa000 0xa2a9aa000(0xa2a9aa000) 00000800:00000200:10.0:1547251679.659003:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ae6ab74000 0xa2ab74000(0xa2ab74000) 00000800:00000200:10.0:1547251679.659004:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b9072d3000 0x14c72d3000(0x14c72d3000) 00000800:00000200:10.0:1547251679.659005:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b81d638000 0x13dd638000(0x13dd638000) 00000800:00000200:10.0:1547251679.659005:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b81d689000 0x13dd689000(0x13dd689000) 00000800:00000200:10.0:1547251679.659006:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b81d40b000 0x13dd40b000(0x13dd40b000) 00000800:00000200:10.0:1547251679.659007:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ae6ab11000 0xa2ab11000(0xa2ab11000) 00000800:00000200:10.0:1547251679.659008:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b907293000 0x14c7293000(0x14c7293000) 00000800:00000200:10.0:1547251679.659009:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b81d787000 0x13dd787000(0x13dd787000) 00000800:00000200:10.0:1547251679.659010:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b81d5ad000 0x13dd5ad000(0x13dd5ad000) 00000800:00000200:10.0:1547251679.659011:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b82461b000 0x13e461b000(0x13e461b000) 00000800:00000200:9.0:1547251679.659603:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659605:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659606:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659607:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659608:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659609:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659610:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659611:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659612:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659613:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659613:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659614:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659615:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659616:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659617:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659618:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659619:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659620:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659620:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659621:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659621:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659622:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659623:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659623:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659624:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659625:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659625:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659626:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659626:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659627:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659628:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659629:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659629:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659630:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:9.0:1547251679.659631:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:9.0:1547251679.659631:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000100:12.0:1547251679.659926:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251679.659929:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251679.659930:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251679.659979:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:12.0:1547251679.659982:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:13.0:1547251679.660000:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (19)++ 00000800:00000200:13.0:1547251679.660006:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (20)-- 00000800:00000200:13.0:1547251679.660007:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (19)-- 00000800:00000200:13.0:1547251679.660008:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (18)-- 00000800:00000200:8.2:1547251679.660110:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07e000] (17)++ 00000800:00000200:13.0:1547251679.660122:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (18)++ 00000800:00000200:13.0:1547251679.660126:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (19)-- 00000800:00000200:13.0:1547251679.660127:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (18)-- 00000800:00000200:13.0:1547251679.660127:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (17)++ 00000800:00000200:13.0:1547251679.660128:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (18)-- 00000800:00000200:13.0:1547251679.660129:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (17)-- 00000800:00000200:13.0:1547251679.660129:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (16)++ 00000800:00000200:13.0:1547251679.660130:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (17)-- 00000800:00000200:13.0:1547251679.660131:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (16)-- 00000800:00000200:13.0:1547251679.660132:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (15)++ 00000800:00000200:13.0:1547251679.660133:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (16)-- 00000800:00000200:13.0:1547251679.660134:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (15)-- 00000800:00000200:15.0:1547251679.660135:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (14)++ 00000800:00000200:15.0:1547251679.660142:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (15)-- 00000800:00000200:15.0:1547251679.660143:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (14)-- 00000800:00000200:15.0:1547251679.660144:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (13)++ 00000800:00000200:15.0:1547251679.660147:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (14)-- 00000800:00000200:15.0:1547251679.660148:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (13)-- 00000800:00000200:15.0:1547251679.660149:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (12)++ 00000800:00000200:15.0:1547251679.660150:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (13)-- 00000800:00000200:13.0:1547251679.660150:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (13)++ 00000800:00000200:15.0:1547251679.660151:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (13)-- 00000800:00000200:15.0:1547251679.660152:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (12)++ 00000800:00000200:13.0:1547251679.660152:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (12)-- 00000800:00000200:13.0:1547251679.660153:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (12)-- 00000800:00000200:15.0:1547251679.660154:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (11)-- 00000800:00000200:13.0:1547251679.660154:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (11)++ 00000800:00000200:15.0:1547251679.660155:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (11)-- 00000800:00000200:15.0:1547251679.660156:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (10)++ 00000800:00000200:13.0:1547251679.660156:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (10)-- 00000800:00000200:13.0:1547251679.660157:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (10)-- 00000800:00000200:15.0:1547251679.660159:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (9)-- 00000800:00000200:13.0:1547251679.660160:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (9)++ 00000800:00000200:15.0:1547251679.660161:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (9)-- 00000800:00000200:13.0:1547251679.660161:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (8)-- 00000800:00000200:15.0:1547251679.660162:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (8)++ 00000800:00000200:13.0:1547251679.660162:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (8)-- 00000800:00000200:15.0:1547251679.660163:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (7)-- 00000800:00000200:15.0:1547251679.660164:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (7)-- 00000800:00000200:13.0:1547251679.660164:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (7)++ 00000800:00000200:15.0:1547251679.660165:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (6)++ 00000800:00000200:13.0:1547251679.660165:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (6)-- 00000800:00000200:13.0:1547251679.660166:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (6)-- 00000800:00000200:15.0:1547251679.660167:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (5)-- 00000800:00000200:13.0:1547251679.660167:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (5)++ 00000800:00000200:15.0:1547251679.660169:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (5)-- 00000800:00000200:13.0:1547251679.660169:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (4)-- 00000800:00000200:15.0:1547251679.660170:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07e000] (4)++ 00000800:00000200:13.0:1547251679.660171:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (4)-- 00000800:00000200:15.0:1547251679.660172:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07e000] (3)-- 00000800:00000200:13.0:1547251679.660172:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (3)-- 00000800:00000200:15.0:1547251679.660173:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07e000] (1)-- 00000800:00000200:9.0:1547251679.660185:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.660940:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251679.660951:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.660960:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251679.660961:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251679.660968:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251679.660977:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251679.661207:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251679.661216:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b630fa6000 0x11f0fa6000(0x11f0fa6000) 00000800:00000200:10.0:1547251679.661217:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97a65d22a000 0x21d22a000(0x21d22a000) 00000800:00000200:10.0:1547251679.661218:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ba92fef000 0x1652fef000(0x1652fef000) 00000800:00000200:10.0:1547251679.661218:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b8247c0000 0x13e47c0000(0x13e47c0000) 00000800:00000200:10.0:1547251679.661219:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ba955fc000 0x16555fc000(0x16555fc000) 00000800:00000200:10.0:1547251679.661220:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba955fd000 0x16555fd000(0x16555fd000) 00000800:00000200:10.0:1547251679.661221:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b7f4704000 0x13b4704000(0x13b4704000) 00000800:00000200:10.0:1547251679.661222:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b7f4705000 0x13b4705000(0x13b4705000) 00000800:00000200:10.0:1547251679.661223:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b17b1a0000 0xd3b1a0000(0xd3b1a0000) 00000800:00000200:10.0:1547251679.661223:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b17b1a1000 0xd3b1a1000(0xd3b1a1000) 00000800:00000200:10.0:1547251679.661224:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97bae1d24000 0x16a1d24000(0x16a1d24000) 00000800:00000200:10.0:1547251679.661225:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97bae1d25000 0x16a1d25000(0x16a1d25000) 00000800:00000200:10.0:1547251679.661226:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b81d7f4000 0x13dd7f4000(0x13dd7f4000) 00000800:00000200:10.0:1547251679.661227:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba9ef48000 0x165ef48000(0x165ef48000) 00000800:00000200:10.0:1547251679.661228:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97baa13a2000 0x16613a2000(0x16613a2000) 00000800:00000200:10.0:1547251679.661228:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba9d3c1000 0x165d3c1000(0x165d3c1000) 00000800:00000200:10.0:1547251679.661229:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ba996e3000 0x16596e3000(0x16596e3000) 00000800:00000200:10.0:1547251679.661230:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97bad6ae0000 0x1696ae0000(0x1696ae0000) 00000800:00000200:9.0:1547251679.661832:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661834:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661835:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661835:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661836:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661836:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661837:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661838:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661838:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661839:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661840:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661840:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661841:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661841:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661842:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661843:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661843:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661844:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661844:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661845:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661846:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661846:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661847:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661847:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661848:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661849:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661849:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661850:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661851:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661851:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661852:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661853:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661853:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661854:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:9.0:1547251679.661854:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:9.0:1547251679.661855:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000100:12.0:1547251679.662090:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251679.662093:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251679.662094:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251679.662136:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:12.0:1547251679.662147:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:13.0:1547251679.662156:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (19)++ 00000800:00000200:13.0:1547251679.662162:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (20)-- 00000800:00000200:13.0:1547251679.662163:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (19)-- 00000800:00000200:13.0:1547251679.662165:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (18)-- 00000800:00000200:8.2:1547251679.662283:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07d000] (17)++ 00000800:00000200:13.0:1547251679.662295:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (18)++ 00000800:00000200:13.0:1547251679.662299:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (19)-- 00000800:00000200:13.0:1547251679.662300:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (18)-- 00000800:00000200:13.0:1547251679.662301:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (17)++ 00000800:00000200:13.0:1547251679.662302:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (18)-- 00000800:00000200:13.0:1547251679.662303:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (17)-- 00000800:00000200:13.0:1547251679.662303:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (16)++ 00000800:00000200:13.0:1547251679.662304:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (17)-- 00000800:00000200:13.0:1547251679.662305:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (16)-- 00000800:00000200:13.0:1547251679.662306:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (15)++ 00000800:00000200:13.0:1547251679.662307:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (16)-- 00000800:00000200:13.0:1547251679.662308:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (15)-- 00000800:00000200:15.0:1547251679.662309:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (14)++ 00000800:00000200:15.0:1547251679.662316:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (15)-- 00000800:00000200:15.0:1547251679.662317:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (14)-- 00000800:00000200:15.0:1547251679.662318:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (13)++ 00000800:00000200:15.0:1547251679.662321:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (14)-- 00000800:00000200:15.0:1547251679.662323:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (13)-- 00000800:00000200:15.0:1547251679.662324:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (12)++ 00000800:00000200:15.0:1547251679.662327:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (13)-- 00000800:00000200:15.0:1547251679.662328:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (12)-- 00000800:00000200:15.0:1547251679.662328:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (11)++ 00000800:00000200:15.0:1547251679.662329:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (12)-- 00000800:00000200:13.0:1547251679.662330:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (12)++ 00000800:00000200:15.0:1547251679.662331:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (12)-- 00000800:00000200:15.0:1547251679.662332:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (11)++ 00000800:00000200:13.0:1547251679.662332:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (11)-- 00000800:00000200:13.0:1547251679.662333:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (11)-- 00000800:00000200:15.0:1547251679.662334:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (10)-- 00000800:00000200:13.0:1547251679.662334:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (10)++ 00000800:00000200:15.0:1547251679.662335:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (10)-- 00000800:00000200:15.0:1547251679.662336:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (9)++ 00000800:00000200:13.0:1547251679.662336:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (9)-- 00000800:00000200:13.0:1547251679.662337:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (9)-- 00000800:00000200:15.0:1547251679.662338:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (8)-- 00000800:00000200:13.0:1547251679.662338:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (8)++ 00000800:00000200:15.0:1547251679.662339:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (8)-- 00000800:00000200:15.0:1547251679.662340:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (7)++ 00000800:00000200:13.0:1547251679.662340:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (7)-- 00000800:00000200:13.0:1547251679.662341:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (7)-- 00000800:00000200:15.0:1547251679.662342:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (6)-- 00000800:00000200:13.0:1547251679.662342:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (6)++ 00000800:00000200:15.0:1547251679.662343:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (6)-- 00000800:00000200:15.0:1547251679.662344:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (5)++ 00000800:00000200:13.0:1547251679.662344:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (5)-- 00000800:00000200:15.0:1547251679.662345:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (6)-- 00000800:00000200:15.0:1547251679.662345:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (5)-- 00000800:00000200:13.0:1547251679.662346:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (3)-- 00000800:00000200:15.0:1547251679.662347:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07d000] (2)++ 00000800:00000200:15.0:1547251679.662350:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07d000] (3)-- 00000800:00000200:15.0:1547251679.662351:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (2)-- 00000800:00000200:15.0:1547251679.662352:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07d000] (1)-- 00000800:00000200:9.0:1547251679.662364:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.663120:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251679.663124:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.663133:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251679.663135:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251679.663141:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251679.663147:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251679.663339:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251679.663345:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b158afd000 0xd18afd000(0xd18afd000) 00000800:00000200:10.0:1547251679.663346:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97badc907000 0x169c907000(0x169c907000) 00000800:00000200:10.0:1547251679.663347:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b073a9d000 0xc33a9d000(0xc33a9d000) 00000800:00000200:10.0:1547251679.663348:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b6fde3e000 0x12bde3e000(0x12bde3e000) 00000800:00000200:10.0:1547251679.663349:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ba92fc4000 0x1652fc4000(0x1652fc4000) 00000800:00000200:10.0:1547251679.663350:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97badc9ea000 0x169c9ea000(0x169c9ea000) 00000800:00000200:10.0:1547251679.663350:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b9073ab000 0x14c73ab000(0x14c73ab000) 00000800:00000200:10.0:1547251679.663352:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97ba95590000 0x1655590000(0x1655590000) 00000800:00000200:10.0:1547251679.663352:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b81d7f5000 0x13dd7f5000(0x13dd7f5000) 00000800:00000200:10.0:1547251679.663353:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b81d4e9000 0x13dd4e9000(0x13dd4e9000) 00000800:00000200:10.0:1547251679.663354:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ae6a83a000 0xa2a83a000(0xa2a83a000) 00000800:00000200:10.0:1547251679.663355:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b82463f000 0x13e463f000(0x13e463f000) 00000800:00000200:10.0:1547251679.663356:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97a65d29d000 0x21d29d000(0x21d29d000) 00000800:00000200:10.0:1547251679.663357:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ad572f9000 0x9172f9000(0x9172f9000) 00000800:00000200:10.0:1547251679.663358:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ae6a970000 0xa2a970000(0xa2a970000) 00000800:00000200:10.0:1547251679.663358:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97a65d341000 0x21d341000(0x21d341000) 00000800:00000200:10.0:1547251679.663359:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b1a46a7000 0xd646a7000(0xd646a7000) 00000800:00000200:10.0:1547251679.663360:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b17b0b9000 0xd3b0b9000(0xd3b0b9000) 00000800:00000200:9.0:1547251679.663928:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663929:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663930:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663930:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663931:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663931:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663932:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663933:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663933:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663934:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663935:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663935:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663936:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663936:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663937:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663938:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663938:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663939:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663939:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663940:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663941:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663941:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663942:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663948:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663949:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663949:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663950:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663950:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663951:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663952:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663952:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663953:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663953:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663954:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:9.0:1547251679.663955:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:9.0:1547251679.663955:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000100:12.0:1547251679.664197:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251679.664200:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251679.664202:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251679.664235:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c600] (19)++ 00000800:00000200:12.0:1547251679.664257:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:13.0:1547251679.664259:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (20)++ 00000800:00000200:13.0:1547251679.664267:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (20)-- 00000800:00000200:13.0:1547251679.664268:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (19)-- 00000800:00000200:13.0:1547251679.664270:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:8.2:1547251679.664376:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c600] (17)++ 00000800:00000200:13.0:1547251679.664388:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)++ 00000800:00000200:13.0:1547251679.664392:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (19)-- 00000800:00000200:13.0:1547251679.664393:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:13.0:1547251679.664394:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (17)++ 00000800:00000200:13.0:1547251679.664395:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (18)-- 00000800:00000200:13.0:1547251679.664395:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (17)-- 00000800:00000200:13.0:1547251679.664396:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (16)++ 00000800:00000200:13.0:1547251679.664397:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (17)-- 00000800:00000200:13.0:1547251679.664398:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (16)-- 00000800:00000200:13.0:1547251679.664399:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)++ 00000800:00000200:13.0:1547251679.664402:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (16)-- 00000800:00000200:13.0:1547251679.664403:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (15)-- 00000800:00000200:13.0:1547251679.664404:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (14)++ 00000800:00000200:13.0:1547251679.664405:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (15)-- 00000800:00000200:13.0:1547251679.664406:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (14)-- 00000800:00000200:15.0:1547251679.664407:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)++ 00000800:00000200:15.0:1547251679.664414:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (14)-- 00000800:00000200:15.0:1547251679.664415:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:15.0:1547251679.664416:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)++ 00000800:00000200:15.0:1547251679.664419:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (13)-- 00000800:00000200:15.0:1547251679.664419:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)-- 00000800:00000200:15.0:1547251679.664420:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)++ 00000800:00000200:15.0:1547251679.664421:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (12)-- 00000800:00000200:15.0:1547251679.664422:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)-- 00000800:00000200:13.0:1547251679.664422:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (12)++ 00000800:00000200:13.0:1547251679.664423:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (11)-- 00000800:00000200:15.0:1547251679.664424:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)++ 00000800:00000200:15.0:1547251679.664425:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (10)-- 00000800:00000200:13.0:1547251679.664425:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (11)-- 00000800:00000200:15.0:1547251679.664426:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (10)-- 00000800:00000200:13.0:1547251679.664426:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (10)++ 00000800:00000200:13.0:1547251679.664427:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (9)-- 00000800:00000200:15.0:1547251679.664428:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (9)++ 00000800:00000200:15.0:1547251679.664429:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:13.0:1547251679.664429:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (9)-- 00000800:00000200:13.0:1547251679.664430:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)++ 00000800:00000200:13.0:1547251679.664430:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (9)-- 00000800:00000200:13.0:1547251679.664431:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (8)-- 00000800:00000200:15.0:1547251679.664432:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (6)-- 00000800:00000200:13.0:1547251679.664433:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)++ 00000800:00000200:13.0:1547251679.664436:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (6)-- 00000800:00000200:13.0:1547251679.664437:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)-- 00000800:00000200:13.0:1547251679.664438:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (4)++ 00000800:00000200:15.0:1547251679.664439:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)++ 00000800:00000200:13.0:1547251679.664439:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (5)-- 00000800:00000200:13.0:1547251679.664440:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (5)-- 00000800:00000200:15.0:1547251679.664441:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (4)-- 00000800:00000200:13.0:1547251679.664441:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c600] (4)++ 00000800:00000200:15.0:1547251679.664442:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (4)-- 00000800:00000200:13.0:1547251679.664443:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c600] (3)-- 00000800:00000200:15.0:1547251679.664444:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (3)-- 00000800:00000200:13.0:1547251679.664449:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c600] (1)-- 00000800:00000200:9.0:1547251679.664460:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.665064:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251679.665068:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.665077:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251679.665078:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251679.665084:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251679.665090:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251679.665562:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251679.665569:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97a99520e000 0x55520e000(0x55520e000) 00000800:00000200:10.0:1547251679.665570:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b81d63e000 0x13dd63e000(0x13dd63e000) 00000800:00000200:10.0:1547251679.665571:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b824611000 0x13e4611000(0x13e4611000) 00000800:00000200:10.0:1547251679.665572:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97b17b2fe000 0xd3b2fe000(0xd3b2fe000) 00000800:00000200:10.0:1547251679.665573:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b17b1a7000 0xd3b1a7000(0xd3b1a7000) 00000800:00000200:10.0:1547251679.665574:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ba9ba4b000 0x165ba4b000(0x165ba4b000) 00000800:00000200:10.0:1547251679.665575:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ba961ae000 0x16561ae000(0x16561ae000) 00000800:00000200:10.0:1547251679.665575:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b80cd1b000 0x13ccd1b000(0x13ccd1b000) 00000800:00000200:10.0:1547251679.665576:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97baa1dbf000 0x1661dbf000(0x1661dbf000) 00000800:00000200:10.0:1547251679.665577:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97badd53d000 0x169d53d000(0x169d53d000) 00000800:00000200:10.0:1547251679.665578:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97badf3ac000 0x169f3ac000(0x169f3ac000) 00000800:00000200:10.0:1547251679.665579:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97baddebc000 0x169debc000(0x169debc000) 00000800:00000200:10.0:1547251679.665580:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97a995224000 0x555224000(0x555224000) 00000800:00000200:10.0:1547251679.665580:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba9cef0000 0x165cef0000(0x165cef0000) 00000800:00000200:10.0:1547251679.665581:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b17b1b1000 0xd3b1b1000(0xd3b1b1000) 00000800:00000200:10.0:1547251679.665582:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b17b399000 0xd3b399000(0xd3b399000) 00000800:00000200:10.0:1547251679.665583:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ae3787e000 0x9f787e000(0x9f787e000) 00000800:00000200:10.0:1547251679.665594:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ae6ab19000 0xa2ab19000(0xa2ab19000) 00000800:00000200:9.0:1547251679.666180:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666182:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666182:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666183:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666183:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666184:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666185:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666185:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666186:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666187:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666187:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666188:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666188:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666189:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666190:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666190:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666191:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666192:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666192:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666193:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666194:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666195:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666195:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666196:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666196:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666197:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666197:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666198:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666199:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666199:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666200:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666200:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666201:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666202:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:9.0:1547251679.666202:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:9.0:1547251679.666203:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000100:12.0:1547251679.666447:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251679.666450:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251679.666452:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251679.666482:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:12.0:1547251679.666502:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:15.0:1547251679.666535:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (19)++ 00000800:00000200:15.0:1547251679.666543:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (20)-- 00000800:00000200:15.0:1547251679.666544:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (19)-- 00000800:00000200:15.0:1547251679.666545:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (18)-- 00000800:00000200:8.2:1547251679.666629:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07cc00] (17)++ 00000800:00000200:15.0:1547251679.666643:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (18)++ 00000800:00000200:15.0:1547251679.666648:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (19)-- 00000800:00000200:15.0:1547251679.666649:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (18)-- 00000800:00000200:15.0:1547251679.666649:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (17)++ 00000800:00000200:15.0:1547251679.666650:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (18)-- 00000800:00000200:15.0:1547251679.666651:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (17)-- 00000800:00000200:15.0:1547251679.666652:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (16)++ 00000800:00000200:15.0:1547251679.666652:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (17)-- 00000800:00000200:15.0:1547251679.666653:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (16)-- 00000800:00000200:15.0:1547251679.666654:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (15)++ 00000800:00000200:15.0:1547251679.666654:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (16)-- 00000800:00000200:15.0:1547251679.666655:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (15)-- 00000800:00000200:15.0:1547251679.666656:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (14)++ 00000800:00000200:15.0:1547251679.666656:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (15)-- 00000800:00000200:15.0:1547251679.666657:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (14)-- 00000800:00000200:13.0:1547251679.666659:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (13)++ 00000800:00000200:13.0:1547251679.666667:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (14)-- 00000800:00000200:13.0:1547251679.666668:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (13)-- 00000800:00000200:15.0:1547251679.666670:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (12)++ 00000800:00000200:15.0:1547251679.666672:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (13)-- 00000800:00000200:15.0:1547251679.666672:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (12)-- 00000800:00000200:15.0:1547251679.666673:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (11)++ 00000800:00000200:15.0:1547251679.666675:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (12)-- 00000800:00000200:15.0:1547251679.666675:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (11)-- 00000800:00000200:15.0:1547251679.666676:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (10)++ 00000800:00000200:15.0:1547251679.666679:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (11)-- 00000800:00000200:15.0:1547251679.666680:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (10)-- 00000800:00000200:15.0:1547251679.666681:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (9)++ 00000800:00000200:15.0:1547251679.666682:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (10)-- 00000800:00000200:13.0:1547251679.666683:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (10)++ 00000800:00000200:15.0:1547251679.666684:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (10)-- 00000800:00000200:15.0:1547251679.666685:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (9)++ 00000800:00000200:13.0:1547251679.666685:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (9)-- 00000800:00000200:13.0:1547251679.666687:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (9)-- 00000800:00000200:15.0:1547251679.666688:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (8)-- 00000800:00000200:13.0:1547251679.666688:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (8)++ 00000800:00000200:15.0:1547251679.666689:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (8)-- 00000800:00000200:13.0:1547251679.666690:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (7)-- 00000800:00000200:15.0:1547251679.666691:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (7)++ 00000800:00000200:15.0:1547251679.666692:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (6)-- 00000800:00000200:13.0:1547251679.666692:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (7)-- 00000800:00000200:13.0:1547251679.666693:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (6)++ 00000800:00000200:15.0:1547251679.666694:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (6)-- 00000800:00000200:15.0:1547251679.666695:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (5)++ 00000800:00000200:13.0:1547251679.666695:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (5)-- 00000800:00000200:15.0:1547251679.666696:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (4)-- 00000800:00000200:13.0:1547251679.666696:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (5)-- 00000800:00000200:13.0:1547251679.666697:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07cc00] (4)++ 00000800:00000200:15.0:1547251679.666698:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (4)-- 00000800:00000200:13.0:1547251679.666698:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07cc00] (3)-- 00000800:00000200:15.0:1547251679.666699:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (3)-- 00000800:00000200:13.0:1547251679.666700:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07cc00] (1)-- 00000800:00000200:9.0:1547251679.666707:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.667438:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251679.667442:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.667451:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251679.667453:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251679.667459:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251679.667465:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251679.668004:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251679.668014:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ae6a9fc000 0xa2a9fc000(0xa2a9fc000) 00000800:00000200:10.0:1547251679.668016:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97a65d0ea000 0x21d0ea000(0x21d0ea000) 00000800:00000200:10.0:1547251679.668017:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ae3782d000 0x9f782d000(0x9f782d000) 00000800:00000200:10.0:1547251679.668018:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97a99502c000 0x55502c000(0x55502c000) 00000800:00000200:10.0:1547251679.668019:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b8246a0000 0x13e46a0000(0x13e46a0000) 00000800:00000200:10.0:1547251679.668020:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97ae6aa3c000 0xa2aa3c000(0xa2aa3c000) 00000800:00000200:10.0:1547251679.668020:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97ae6aaa5000 0xa2aaa5000(0xa2aaa5000) 00000800:00000200:10.0:1547251679.668021:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b82461b000 0x13e461b000(0x13e461b000) 00000800:00000200:10.0:1547251679.668022:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97ba90fc2000 0x1650fc2000(0x1650fc2000) 00000800:00000200:10.0:1547251679.668023:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97ba90fc3000 0x1650fc3000(0x1650fc3000) 00000800:00000200:10.0:1547251679.668024:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97ba9727a000 0x165727a000(0x165727a000) 00000800:00000200:10.0:1547251679.668025:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97ba9727b000 0x165727b000(0x165727b000) 00000800:00000200:10.0:1547251679.668026:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b1233de000 0xce33de000(0xce33de000) 00000800:00000200:10.0:1547251679.668026:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97b1233df000 0xce33df000(0xce33df000) 00000800:00000200:10.0:1547251679.668027:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ba8cbba000 0x164cbba000(0x164cbba000) 00000800:00000200:10.0:1547251679.668028:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97ba8cbbb000 0x164cbbb000(0x164cbbb000) 00000800:00000200:10.0:1547251679.668029:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b46b420000 0x102b420000(0x102b420000) 00000800:00000200:10.0:1547251679.668030:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97b46b421000 0x102b421000(0x102b421000) 00000800:00000200:9.0:1547251679.668607:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668609:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668610:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668611:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668612:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668613:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668614:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668615:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668616:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668617:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668617:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668618:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668619:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668620:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668620:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668622:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668622:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668623:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668624:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668624:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668625:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668625:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668626:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668627:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668627:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668628:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668629:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668629:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668630:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668630:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668631:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668632:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668632:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668633:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:9.0:1547251679.668634:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:9.0:1547251679.668634:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000100:12.0:1547251679.668903:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251679.668907:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251679.668908:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251679.668942:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:12.0:1547251679.668967:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:15.0:1547251679.669009:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (19)++ 00000800:00000200:15.0:1547251679.669017:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (20)-- 00000800:00000200:15.0:1547251679.669018:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (19)-- 00000800:00000200:15.0:1547251679.669020:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (18)-- 00000800:00000200:8.2:1547251679.669082:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07c800] (17)++ 00000800:00000200:15.0:1547251679.669093:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (18)++ 00000800:00000200:15.0:1547251679.669098:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (19)-- 00000800:00000200:15.0:1547251679.669098:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (18)-- 00000800:00000200:15.0:1547251679.669099:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (17)++ 00000800:00000200:15.0:1547251679.669100:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (18)-- 00000800:00000200:15.0:1547251679.669101:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (17)-- 00000800:00000200:15.0:1547251679.669101:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (16)++ 00000800:00000200:15.0:1547251679.669102:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (17)-- 00000800:00000200:15.0:1547251679.669103:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (16)-- 00000800:00000200:15.0:1547251679.669103:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (15)++ 00000800:00000200:15.0:1547251679.669105:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (16)-- 00000800:00000200:15.0:1547251679.669105:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (15)-- 00000800:00000200:13.0:1547251679.669107:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (14)++ 00000800:00000200:13.0:1547251679.669113:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (15)-- 00000800:00000200:13.0:1547251679.669114:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (14)-- 00000800:00000200:13.0:1547251679.669115:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (13)++ 00000800:00000200:13.0:1547251679.669118:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (14)-- 00000800:00000200:13.0:1547251679.669118:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (13)-- 00000800:00000200:13.0:1547251679.669119:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (12)++ 00000800:00000200:13.0:1547251679.669120:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (13)-- 00000800:00000200:15.0:1547251679.669121:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (13)++ 00000800:00000200:13.0:1547251679.669122:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (13)-- 00000800:00000200:15.0:1547251679.669123:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (12)-- 00000800:00000200:13.0:1547251679.669123:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (12)++ 00000800:00000200:15.0:1547251679.669124:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (12)-- 00000800:00000200:15.0:1547251679.669125:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (11)++ 00000800:00000200:13.0:1547251679.669125:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (11)-- 00000800:00000200:13.0:1547251679.669126:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (11)-- 00000800:00000200:15.0:1547251679.669127:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (10)-- 00000800:00000200:13.0:1547251679.669127:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (10)++ 00000800:00000200:15.0:1547251679.669128:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (10)-- 00000800:00000200:15.0:1547251679.669129:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (9)++ 00000800:00000200:13.0:1547251679.669129:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (9)-- 00000800:00000200:13.0:1547251679.669130:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (9)-- 00000800:00000200:15.0:1547251679.669131:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (8)-- 00000800:00000200:13.0:1547251679.669131:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (8)++ 00000800:00000200:15.0:1547251679.669132:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (8)-- 00000800:00000200:15.0:1547251679.669133:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (7)++ 00000800:00000200:13.0:1547251679.669133:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (7)-- 00000800:00000200:13.0:1547251679.669134:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (7)-- 00000800:00000200:15.0:1547251679.669135:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (6)-- 00000800:00000200:13.0:1547251679.669135:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (6)++ 00000800:00000200:15.0:1547251679.669136:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (6)-- 00000800:00000200:15.0:1547251679.669138:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (5)++ 00000800:00000200:13.0:1547251679.669138:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (5)-- 00000800:00000200:13.0:1547251679.669140:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (5)-- 00000800:00000200:15.0:1547251679.669143:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (4)-- 00000800:00000200:13.0:1547251679.669143:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07c800] (4)++ 00000800:00000200:15.0:1547251679.669144:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (4)-- 00000800:00000200:13.0:1547251679.669145:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07c800] (3)-- 00000800:00000200:15.0:1547251679.669146:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (3)-- 00000800:00000200:13.0:1547251679.669147:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07c800] (1)-- 00000800:00000200:9.0:1547251679.669159:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.669909:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:9.0:1547251679.669913:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.669922:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:9.0:1547251679.669923:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:10.0:1547251679.669929:0:15028:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000800:00000200:10.0:1547251679.669937:0:15028:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000800:00000200:10.0:1547251679.670260:0:15028:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:10.0:1547251679.670266:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97ba91ae1000 0x1651ae1000(0x1651ae1000) 00000800:00000200:10.0:1547251679.670267:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b4e7eda000 0x10a7eda000(0x10a7eda000) 00000800:00000200:10.0:1547251679.670268:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97b4e7edb000 0x10a7edb000(0x10a7edb000) 00000800:00000200:10.0:1547251679.670269:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ba8774e000 0x164774e000(0x164774e000) 00000800:00000200:10.0:1547251679.670270:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97ba8774f000 0x164774f000(0x164774f000) 00000800:00000200:10.0:1547251679.670270:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b81d5da000 0x13dd5da000(0x13dd5da000) 00000800:00000200:10.0:1547251679.670272:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97b81d5db000 0x13dd5db000(0x13dd5db000) 00000800:00000200:10.0:1547251679.670273:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97b17b2e4000 0xd3b2e4000(0xd3b2e4000) 00000800:00000200:10.0:1547251679.670274:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b17b2e5000 0xd3b2e5000(0xd3b2e5000) 00000800:00000200:10.0:1547251679.670275:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97a744b58000 0x304b58000(0x304b58000) 00000800:00000200:10.0:1547251679.670275:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97a744b59000 0x304b59000(0x304b59000) 00000800:00000200:10.0:1547251679.670276:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b17b0bc000 0xd3b0bc000(0xd3b0bc000) 00000800:00000200:10.0:1547251679.670277:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97b17b0bd000 0xd3b0bd000(0xd3b0bd000) 00000800:00000200:10.0:1547251679.670278:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ac1b2d0000 0x7db2d0000(0x7db2d0000) 00000800:00000200:10.0:1547251679.670279:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97ac1b2d1000 0x7db2d1000(0x7db2d1000) 00000800:00000200:10.0:1547251679.670280:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b17b3ee000 0xd3b3ee000(0xd3b3ee000) 00000800:00000200:10.0:1547251679.670281:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97b17b3ef000 0xd3b3ef000(0xd3b3ef000) 00000800:00000200:10.0:1547251679.670282:0:15028:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ae6a8f0000 0xa2a8f0000(0xa2a8f0000) 00000800:00000200:9.0:1547251679.670855:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670857:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670857:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670858:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670859:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670859:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670860:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670860:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670861:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670862:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670862:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670864:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670864:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670865:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670865:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670866:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670867:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670867:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670868:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670868:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670869:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670870:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670870:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670871:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670871:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670872:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670873:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670873:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670874:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670874:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670875:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670876:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670876:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670877:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:9.0:1547251679.670878:0:15028:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:9.0:1547251679.670878:0:15028:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000100:12.0:1547251679.671184:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251679.671187:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251679.671189:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:8.2:1547251679.671245:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:12.0:1547251679.671255:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:15.0:1547251679.671297:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (19)++ 00000800:00000200:15.0:1547251679.671306:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (20)-- 00000800:00000200:15.0:1547251679.671307:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (19)-- 00000800:00000200:15.0:1547251679.671308:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (18)-- 00000800:00000200:8.2:1547251679.671357:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a65d07f200] (17)++ 00000800:00000200:15.0:1547251679.671369:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (18)++ 00000800:00000200:15.0:1547251679.671374:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (19)-- 00000800:00000200:15.0:1547251679.671374:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (18)-- 00000800:00000200:15.0:1547251679.671375:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (17)++ 00000800:00000200:15.0:1547251679.671376:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (18)-- 00000800:00000200:15.0:1547251679.671377:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (17)-- 00000800:00000200:15.0:1547251679.671377:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (16)++ 00000800:00000200:15.0:1547251679.671378:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (17)-- 00000800:00000200:15.0:1547251679.671379:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (16)-- 00000800:00000200:15.0:1547251679.671380:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (15)++ 00000800:00000200:15.0:1547251679.671383:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (16)-- 00000800:00000200:15.0:1547251679.671384:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (15)-- 00000800:00000200:15.0:1547251679.671385:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (14)++ 00000800:00000200:15.0:1547251679.671388:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (15)-- 00000800:00000200:15.0:1547251679.671388:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (14)-- 00000800:00000200:15.0:1547251679.671389:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (13)++ 00000800:00000200:15.0:1547251679.671390:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (14)-- 00000800:00000200:15.0:1547251679.671391:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (13)-- 00000800:00000200:13.0:1547251679.671392:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (12)++ 00000800:00000200:13.0:1547251679.671398:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (13)-- 00000800:00000200:13.0:1547251679.671399:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (12)-- 00000800:00000200:13.0:1547251679.671400:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (11)++ 00000800:00000200:13.0:1547251679.671403:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (12)-- 00000800:00000200:13.0:1547251679.671403:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (11)-- 00000800:00000200:13.0:1547251679.671404:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (10)++ 00000800:00000200:15.0:1547251679.671405:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (11)++ 00000800:00000200:13.0:1547251679.671405:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (11)-- 00000800:00000200:13.0:1547251679.671407:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (11)-- 00000800:00000200:15.0:1547251679.671408:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (10)-- 00000800:00000200:13.0:1547251679.671408:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (10)++ 00000800:00000200:15.0:1547251679.671409:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (10)-- 00000800:00000200:15.0:1547251679.671410:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (9)++ 00000800:00000200:13.0:1547251679.671410:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (9)-- 00000800:00000200:13.0:1547251679.671411:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (9)-- 00000800:00000200:15.0:1547251679.671412:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (8)-- 00000800:00000200:13.0:1547251679.671412:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (8)++ 00000800:00000200:15.0:1547251679.671413:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (8)-- 00000800:00000200:13.0:1547251679.671413:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (7)-- 00000800:00000200:15.0:1547251679.671414:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (7)++ 00000800:00000200:13.0:1547251679.671415:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (7)-- 00000800:00000200:15.0:1547251679.671416:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (6)-- 00000800:00000200:13.0:1547251679.671416:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (6)++ 00000800:00000200:15.0:1547251679.671417:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (6)-- 00000800:00000200:15.0:1547251679.671418:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (5)++ 00000800:00000200:13.0:1547251679.671418:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (5)-- 00000800:00000200:13.0:1547251679.671419:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (5)-- 00000800:00000200:15.0:1547251679.671420:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (4)-- 00000800:00000200:13.0:1547251679.671420:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a65d07f200] (4)++ 00000800:00000200:13.0:1547251679.671421:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97a65d07f200] (5)-- 00000800:00000200:15.0:1547251679.671422:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (2)-- 00000800:00000200:13.0:1547251679.671422:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (4)-- 00000800:00000200:13.0:1547251679.671423:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a65d07f200] (1)-- 00000800:00000200:9.0:1547251679.671435:0:12658:0:(o2iblnd_cb.c:3501:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:9.0:1547251679.672257:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000200:0.2:1547251680.093369:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:12.2:1547251680.093378:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:4.0:1547251680.093429:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251680.093433:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251680.093433:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000800:00000200:5.0:1547251680.093436:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251680.093439:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251680.093441:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251680.093445:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b10350 00000400:00000200:4.0:1547251680.093445:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b10360 00000400:00000200:4.0:1547251680.093449:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 64512 00000400:00000200:5.0:1547251680.093454:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 64736 00000100:00000200:4.0:1547251680.093457:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000100:00000200:5.0:1547251680.093468:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:4.0:1547251680.093468:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251680.093470:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251680.093472:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:4.0:1547251680.093473:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251680.093475:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000400:00000200:5.0:1547251680.093476:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251680.093477:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000800:00000200:5.0:1547251680.093478:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251680.093480:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251680.093481:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000400:00000200:4.0:1547251680.093481:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:5.0:1547251680.093483:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251680.093485:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251680.093485:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 26 MB=0x5c38cb6b10340 00000400:00000200:4.0:1547251680.093488:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0xd5649 [1] + 0 00000400:00000200:5.0:1547251680.093489:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251680.093491:0:12661:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d52daa0 00000400:00000200:5.0:1547251680.093493:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 224 into portal 12 MB=0x5c38cb6b10370 00000100:00000200:4.0:1547251680.093493:0:12661:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:5.0:1547251680.093497:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.92@o2ib10 of length 224/224 into md 0x515 [1] + 64960 00000100:00000200:5.0:1547251680.093500:0:12662:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:4.0:1547251680.093500:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d52daa0 00000400:00000200:4.0:1547251680.093501:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251680.093503:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000400:00000200:5.0:1547251680.093504:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251680.093505:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:5.0:1547251680.093506:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251680.093506:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251680.093507:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:4.0:1547251680.093508:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:5.0:1547251680.093509:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251680.093510:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000100:00000200:5.0:1547251680.093528:0:12980:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011445088 00000100:00000200:6.0:1547251680.093532:0:13338:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011445104 00000100:00000200:7.0:1547251680.093534:0:13355:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011445072 00010000:00000200:5.0:1547251680.093534:0:12980:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444bf00 x1622384011445088/t0(0) o400->cache1-MDT0000-lwp-OST0006_UUID@10.0.11.92@o2ib10:741/0 lens 224/224 e 0 to 0 dl 1547251686 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:7.0:1547251680.093540:0:13355:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae5c433f00 x1622384011445072/t0(0) o400->cache1-MDT0000-lwp-OST0007_UUID@10.0.11.92@o2ib10:741/0 lens 224/224 e 0 to 0 dl 1547251686 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:6.0:1547251680.093540:0:13338:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae5c430300 x1622384011445104/t0(0) o400->cache1-MDT0000-lwp-OST0008_UUID@10.0.11.92@o2ib10:741/0 lens 224/224 e 0 to 0 dl 1547251686 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:5.0:1547251680.093543:0:12980:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011445088, offset 192 00000400:00000200:5.0:1547251680.093546:0:12980:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000100:00000200:7.0:1547251680.093549:0:13355:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011445072, offset 192 00000100:00000200:6.0:1547251680.093549:0:13338:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384011445104, offset 192 00000400:00000200:5.0:1547251680.093551:0:12980:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:6.0:1547251680.093553:0:13338:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000400:00000200:5.0:1547251680.093554:0:12980:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251680.093556:0:13355:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.92@o2ib10 00000400:00000200:5.0:1547251680.093556:0:12980:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:5.0:1547251680.093558:0:12980:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9768, 9768] 00000100:00000200:2.0:1547251680.093559:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384011445056 00000400:00000200:5.0:1547251680.093561:0:12980:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00010000:00000200:2.0:1547251680.093565:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692fe9450 x1622384011445056/t0(0) o400->04d65b15-cf9c-7ded-611d-ffe7fe3f3abf@10.0.11.92@o2ib10:741/0 lens 224/224 e 0 to 0 dl 1547251686 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:5.0:1547251680.093567:0:12980:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:5.0:1547251680.093570:0:12980:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:6.0:1547251680.093572:0:13338:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00000100:00000200:2.0:1547251680.093575:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384011445056, offset 192 00000400:00000200:6.0:1547251680.093577:0:13338:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251680.093578:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:6.0:1547251680.093579:0:13338:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251680.093580:0:13338:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9768, 9769] 00000400:00000200:2.0:1547251680.093582:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:2.0:1547251680.093594:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251680.093595:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251680.093597:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9768, 9769] 00000400:00000200:2.0:1547251680.093598:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:6.0:1547251680.093600:0:13338:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:6.0:1547251680.093609:0:13338:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.1.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:6.0:1547251680.093613:0:13338:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:7.0:1547251680.093614:0:13355:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.92@o2ib10 local destination 00000400:00000200:7.0:1547251680.093617:0:13355:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251680.093619:0:13355:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:7.0:1547251680.093621:0:13355:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9770, 9769] 00000400:00000200:7.0:1547251680.093623:0:13355:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:7.0:1547251680.093630:0:13355:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.92@o2ib10(10.1.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:7.0:1547251680.093634:0:13355:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:5.0:1547251680.093636:0:12980:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:5.0:1547251680.093637:0:12980:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:5.0:1547251680.093638:0:12980:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000400:00000200:2.0:1547251680.093638:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:5.0:1547251680.093640:0:12980:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:2.0:1547251680.093641:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:7.0:1547251680.093643:0:13355:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:6.0:1547251680.093643:0:13338:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:7.0:1547251680.093645:0:13355:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:6.0:1547251680.093645:0:13338:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000800:00000200:2.0:1547251680.093645:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:7.0:1547251680.093646:0:13355:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:6.0:1547251680.093646:0:13338:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000800:00000200:2.0:1547251680.093646:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:2.0:1547251680.093647:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:7.0:1547251680.093650:0:13355:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:6.0:1547251680.093650:0:13338:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:2.0:1547251680.093650:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:0.2:1547251680.093652:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000100:00000200:2.0:1547251680.093654:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:0.2:1547251680.093662:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251680.093668:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:5.0:1547251680.093671:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000400:00000200:5.0:1547251680.093673:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad330 00000400:00000200:5.0:1547251680.093674:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251680.093677:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251680.093679:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251680.093680:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251680.093680:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251680.093681:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:5.0:1547251680.093683:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:4.0:1547251680.093683:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b630fe9000 00000400:00000200:4.0:1547251680.093684:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251680.093687:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251680.093689:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:12.2:1547251680.093706:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000800:00000200:12.2:1547251680.093716:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251680.093723:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251680.093724:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000800:00000200:4.0:1547251680.093724:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251680.093725:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000400:00000200:4.0:1547251680.093727:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea330 00000400:00000200:5.0:1547251680.093728:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990c38 00000400:00000200:4.0:1547251680.093728:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251680.093729:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251680.093731:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:5.0:1547251680.093732:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251680.093733:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251680.093734:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251680.093735:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:5.0:1547251680.093736:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:7.2:1547251680.127711:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:9.2:1547251680.127725:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:15.0:1547251680.127768:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:15.0:1547251680.127772:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251680.127775:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251680.127779:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510350 00000400:00000200:15.0:1547251680.127783:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 31040 00000800:00000200:13.0:1547251680.127783:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000100:00000200:15.0:1547251680.127786:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:13.0:1547251680.127787:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:13.0:1547251680.127790:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251680.127794:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510360 00000400:00000200:15.0:1547251680.127796:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251680.127798:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000400:00000200:13.0:1547251680.127798:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 31264 00000800:00000200:15.0:1547251680.127800:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:15.0:1547251680.127801:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:15.0:1547251680.127802:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:13.0:1547251680.127802:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:15.0:1547251680.127804:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251680.127807:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251680.127809:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510370 00000400:00000200:13.0:1547251680.127810:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:15.0:1547251680.127811:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0x4cd [1] + 31488 00000100:00000200:15.0:1547251680.127813:0:12666:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:13.0:1547251680.127813:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:13.0:1547251680.127815:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:13.0:1547251680.127816:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:15.0:1547251680.127819:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251680.127820:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:13.0:1547251680.127820:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251680.127821:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:15.0:1547251680.127822:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:15.0:1547251680.127823:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:13.0:1547251680.127823:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000800:00000200:15.0:1547251680.127825:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:13.0:1547251680.127827:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000100:00000200:15.0:1547251680.127842:0:13371:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821968 00000100:00000200:12.0:1547251680.127844:0:12987:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821984 00000100:00000200:13.0:1547251680.127846:0:13337:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954822000 00010000:00000200:15.0:1547251680.127850:0:13371:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b81d4a5700 x1622383954821968/t0(0) o400->cache1-MDT0000-lwp-OST000a_UUID@10.0.11.93@o2ib10:741/0 lens 224/224 e 0 to 0 dl 1547251686 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:13.0:1547251680.127851:0:13337:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b81d4a1500 x1622383954822000/t0(0) o400->cache1-MDT0000-lwp-OST000b_UUID@10.0.11.93@o2ib10:741/0 lens 224/224 e 0 to 0 dl 1547251686 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:12.0:1547251680.127851:0:12987:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b6d04f2700 x1622383954821984/t0(0) o400->cache1-MDT0000-lwp-OST0009_UUID@10.0.11.93@o2ib10:741/0 lens 224/224 e 0 to 0 dl 1547251686 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:15.0:1547251680.127858:0:13371:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821968, offset 192 00000100:00000200:13.0:1547251680.127860:0:13337:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954822000, offset 192 00000100:00000200:12.0:1547251680.127860:0:12987:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954821984, offset 192 00000400:00000200:15.0:1547251680.127862:0:13371:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000400:00000200:13.0:1547251680.127865:0:13337:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.93@o2ib10 00000400:00000200:15.0:1547251680.127866:0:13371:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00000400:00000200:12.0:1547251680.127868:0:12987:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:15.0:1547251680.127869:0:13371:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251680.127871:0:13371:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251680.127873:0:13371:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9768, 9769] 00000400:00000200:15.0:1547251680.127876:0:13371:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251680.127881:0:13371:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.1.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251680.127884:0:13371:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:13.0:1547251680.127885:0:13337:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.93@o2ib10 local destination 00000400:00000200:13.0:1547251680.127888:0:13337:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251680.127890:0:13337:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251680.127892:0:13337:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9769, 9769] 00000400:00000200:13.0:1547251680.127894:0:13337:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:13.0:1547251680.127901:0:13337:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.1.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251680.127905:0:13337:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:12.0:1547251680.127906:0:12987:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:12.0:1547251680.127909:0:12987:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251680.127911:0:12987:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251680.127912:0:12987:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9769, 9770] 00000400:00000200:12.0:1547251680.127914:0:12987:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:12.0:1547251680.127920:0:12987:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251680.127924:0:12987:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:15.0:1547251680.127925:0:13371:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251680.127926:0:13371:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:13.0:1547251680.127926:0:13337:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251680.127927:0:13371:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:12.0:1547251680.127927:0:12987:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251680.127928:0:13337:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000800:00000200:15.0:1547251680.127929:0:13371:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:13.0:1547251680.127929:0:13337:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:12.0:1547251680.127929:0:12987:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:12.0:1547251680.127930:0:12987:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:13.0:1547251680.127932:0:13337:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000800:00000200:12.0:1547251680.127932:0:12987:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:7.2:1547251680.127942:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:13.0:1547251680.127954:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 224 into portal 26 MB=0x5c38cb3510340 00000800:00000200:9.2:1547251680.127955:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:7.2:1547251680.127957:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000400:00000200:13.0:1547251680.127958:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.93@o2ib10 of length 224/224 into md 0xd5659 [1] + 0 00000800:00000200:15.0:1547251680.127959:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:15.0:1547251680.127960:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000400:00000200:13.0:1547251680.127961:0:12665:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b17b06a000 00000400:00000200:15.0:1547251680.127962:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb15d8 00000400:00000200:15.0:1547251680.127962:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:13.0:1547251680.127963:0:12665:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:15.0:1547251680.127965:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:15.0:1547251680.127967:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:15.0:1547251680.127968:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:15.0:1547251680.127969:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000400:00000200:15.0:1547251680.127970:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038dd0 00000400:00000200:15.0:1547251680.127971:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251680.127972:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a000 00000400:00000200:15.0:1547251680.127973:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000400:00000200:13.0:1547251680.127973:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251680.127974:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:13.0:1547251680.127975:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251680.127976:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:15.0:1547251680.127977:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000800:00000200:13.0:1547251680.127977:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)-- 00000400:00000200:15.0:1547251680.127978:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b2a8 00000400:00000200:15.0:1547251680.127978:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:13.0:1547251680.127978:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000400:00000200:15.0:1547251680.127980:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251680.127980:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:13.0:1547251680.127981:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:15.0:1547251680.127982:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:13.0:1547251680.127983:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:2.0:1547251680.128033:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954821952 00010000:00000200:2.0:1547251680.128039:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692fee050 x1622383954821952/t0(0) o400->b99b3dbb-ff3e-ba67-561d-3f0ab43da035@10.0.11.93@o2ib10:741/0 lens 224/224 e 0 to 0 dl 1547251686 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:2.0:1547251680.128046:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954821952, offset 192 00000400:00000200:2.0:1547251680.128048:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:2.0:1547251680.128052:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:2.0:1547251680.128054:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251680.128056:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251680.128057:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9770, 9770] 00000400:00000200:2.0:1547251680.128059:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:2.0:1547251680.128064:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251680.128067:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:2.0:1547251680.128070:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251680.128071:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:2.0:1547251680.128072:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:2.0:1547251680.128075:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000100:00000200:2.0:1547251680.128079:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000800:00000200:9.2:1547251680.128086:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251680.128100:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:15.0:1547251680.128106:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000400:00000200:15.0:1547251680.128108:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b630fe9110 00000400:00000200:15.0:1547251680.128109:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:15.0:1547251680.128111:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:15.0:1547251680.128113:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:15.0:1547251680.128114:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:5.2:1547251680.159428:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251680.159438:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:10.0:1547251680.159488:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251680.159492:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251680.159493:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251680.159495:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251680.159498:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251680.159500:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251680.159503:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 26 MB=0x5c38cb3510370 00000400:00000200:11.0:1547251680.159504:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb35103a0 00000400:00000200:10.0:1547251680.159507:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0xd5669 [1] + 0 00000400:00000200:10.0:1547251680.159509:0:12664:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b17b06a110 00000400:00000200:11.0:1547251680.159513:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 14784 00000100:00000200:10.0:1547251680.159516:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:10.0:1547251680.159524:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a110 00000100:00000200:11.0:1547251680.159526:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:10.0:1547251680.159526:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251680.159528:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251680.159530:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251680.159531:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251680.159533:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:11.0:1547251680.159535:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251680.159536:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:11.0:1547251680.159537:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251680.159539:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251680.159540:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:10.0:1547251680.159540:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:11.0:1547251680.159542:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000400:00000200:10.0:1547251680.159543:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510390 00000800:00000200:11.0:1547251680.159544:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251680.159547:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 15008 00000400:00000200:11.0:1547251680.159549:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000100:00000200:10.0:1547251680.159549:0:12664:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:11.0:1547251680.159553:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 224 into portal 12 MB=0x5c38cb3510380 00000400:00000200:10.0:1547251680.159554:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:11.0:1547251680.159556:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-10.0.11.91@o2ib10 of length 224/224 into md 0x8a9 [1] + 15232 00000800:00000200:10.0:1547251680.159556:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251680.159557:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251680.159558:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000100:00000200:11.0:1547251680.159559:0:12663:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000800:00000200:10.0:1547251680.159560:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000400:00000200:11.0:1547251680.159562:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251680.159562:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251680.159563:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251680.159565:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251680.159566:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:11.0:1547251680.159582:0:12983:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954822048 00000100:00000200:2.0:1547251680.159593:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954822000 00000100:00000200:10.0:1547251680.159596:0:13336:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954822032 00010000:00000200:11.0:1547251680.159599:0:12983:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ba960eb900 x1622383954822048/t0(0) o400->cache1-MDT0000-lwp-OST0004_UUID@10.0.11.91@o2ib10:741/0 lens 224/224 e 0 to 0 dl 1547251686 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:9.0:1547251680.159599:0:13370:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622383954822016 00010000:00000200:2.0:1547251680.159599:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a692fecc50 x1622383954822000/t0(0) o400->dc7951f7-3e71-e9c4-2e5e-5f95f4837d68@10.0.11.91@o2ib10:741/0 lens 224/224 e 0 to 0 dl 1547251686 ref 1 fl Interpret:/0/0 rc 0/0 00010000:00000200:10.0:1547251680.159601:0:13336:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b17b07da00 x1622383954822032/t0(0) o400->cache1-MDT0000-lwp-OST0003_UUID@10.0.11.91@o2ib10:741/0 lens 224/224 e 0 to 0 dl 1547251686 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:2.0:1547251680.159606:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622383954822000, offset 192 00000100:00000200:11.0:1547251680.159608:0:12983:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954822048, offset 192 00000100:00000200:10.0:1547251680.159608:0:13336:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954822032, offset 192 00010000:00000200:9.0:1547251680.159608:0:13370:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ba960ef500 x1622383954822016/t0(0) o400->cache1-MDT0000-lwp-OST0005_UUID@10.0.11.91@o2ib10:741/0 lens 224/224 e 0 to 0 dl 1547251686 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:2.0:1547251680.159608:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:11.0:1547251680.159611:0:12983:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000400:00000200:2.0:1547251680.159612:0:12929:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:10.0:1547251680.159614:0:13336:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:2.0:1547251680.159614:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251680.159615:0:12983:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000400:00000200:2.0:1547251680.159616:0:12929:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:9.0:1547251680.159617:0:13370:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622383954822016, offset 192 00000400:00000200:11.0:1547251680.159618:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251680.159618:0:12929:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9775, 9774] 00000400:00000200:11.0:1547251680.159620:0:12983:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251680.159620:0:12929:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:9.0:1547251680.159621:0:13370:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.1.11.91@o2ib10 00000400:00000200:11.0:1547251680.159622:0:12983:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9775, 9775] 00000400:00000200:11.0:1547251680.159624:0:12983:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251680.159631:0:12983:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.1.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251680.159635:0:12983:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:10.0:1547251680.159636:0:13336:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.1.11.90@o2ib10 to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:10.0:1547251680.159638:0:13336:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251680.159640:0:13336:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251680.159642:0:13336:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9775, 9776] 00000400:00000200:10.0:1547251680.159644:0:13336:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251680.159650:0:13336:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:10.0:1547251680.159653:0:13336:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:2.0:1547251680.159657:0:12929:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:10.1.11.90@o2ib10) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251680.159660:0:12929:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:9.0:1547251680.159661:0:13370:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source Specified: 10.0.11.90@o2ib10 to MR: 10.1.11.91@o2ib10 local destination 00000800:00000200:2.0:1547251680.159663:0:12929:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000400:00000200:9.0:1547251680.159664:0:13370:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:2.0:1547251680.159664:0:12929:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000400:00000200:9.0:1547251680.159665:0:13370:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:2.0:1547251680.159665:0:12929:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:9.0:1547251680.159666:0:13370:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 6], s:[9776, 9776] 00000800:00000200:2.0:1547251680.159668:0:12929:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:9.0:1547251680.159669:0:13370:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000100:00000200:2.0:1547251680.159672:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000400:00000200:9.0:1547251680.159674:0:13370:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:10.0.11.90@o2ib10) -> 10.1.11.91@o2ib10(10.1.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251680.159677:0:13370:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 192 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:11.0:1547251680.159679:0:12983:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:10.0:1547251680.159679:0:13336:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:7.2:1547251680.159679:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:9.0:1547251680.159680:0:13370:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251680.159681:0:12983:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:10.0:1547251680.159681:0:13336:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:11.0:1547251680.159682:0:12983:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251680.159682:0:13336:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:9.0:1547251680.159682:0:13370:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:9.0:1547251680.159683:0:13370:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251680.159685:0:12983:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251680.159685:0:13336:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000800:00000200:9.0:1547251680.159686:0:13370:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251680.159695:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251680.159704:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:7.2:1547251680.159704:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000800:00000200:5.2:1547251680.159706:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:5.2:1547251680.159707:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00000400:00000200:10.0:1547251680.159708:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b630fe9220 00000400:00000200:10.0:1547251680.159710:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251680.159712:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:10.0:1547251680.159713:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251680.159714:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251680.159715:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251680.159715:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000400:00000200:11.0:1547251680.159717:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a9145d8 00000800:00000200:10.0:1547251680.159717:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000400:00000200:11.0:1547251680.159718:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251680.159718:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000400:00000200:10.0:1547251680.159720:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b908 00000400:00000200:11.0:1547251680.159721:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251680.159721:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251680.159723:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:10.0:1547251680.159724:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251680.159725:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251680.159727:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000800:00000200:10.0:1547251680.159727:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:11.0:1547251680.159729:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaee0 00000800:00000200:10.0:1547251680.159729:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000400:00000200:11.0:1547251680.159730:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:10.0:1547251680.159730:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:10.0:1547251680.159732:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:11.0:1547251680.159733:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251680.159735:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:11.0:1547251680.186678:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447840, portal 4 00000100:00000200:11.0:1547251680.186681:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447840, offset 0 00000400:00000200:11.0:1547251680.186684:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:11.0:1547251680.186687:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:11.0:1547251680.186689:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23095] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:11.0:1547251680.186692:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23095] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23095] 00000400:00000200:11.0:1547251680.186694:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:11.0:1547251680.186696:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251680.186697:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251680.186699:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9770, 9771] 00000400:00000200:11.0:1547251680.186700:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:11.0:1547251680.186705:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251680.186708:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:11.0:1547251680.186711:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251680.186712:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:11.0:1547251680.186713:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:11.0:1547251680.186716:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:7.2:1547251680.186730:0:13041:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:6.0:1547251680.186749:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447856, portal 4 00000100:00000200:6.0:1547251680.186753:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447856, offset 0 00000400:00000200:6.0:1547251680.186757:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:6.0:1547251680.186761:0:12677:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:6.0:1547251680.186763:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23096] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:6.0:1547251680.186766:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23095] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23096] 00000400:00000200:6.0:1547251680.186769:0:12677:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:6.0:1547251680.186770:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251680.186772:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251680.186773:0:12677:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9771, 9771] 00000400:00000200:6.0:1547251680.186775:0:12677:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:6.0:1547251680.186779:0:12677:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:6.0:1547251680.186782:0:12677:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:13.0:1547251680.186784:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:6.0:1547251680.186785:0:12677:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:6.0:1547251680.186786:0:12677:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:6.0:1547251680.186788:0:12677:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:6.0:1547251680.186790:0:12677:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:13.0:1547251680.186793:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:13.0:1547251680.186798:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07e900 x1622384012447840/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251687 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251680.186804:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b4c8 00000400:00000200:13.0:1547251680.186805:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:9.2:1547251680.186805:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000400:00000200:13.0:1547251680.186808:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251680.186810:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000100:00000200:0.0:1547251680.186810:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447872, portal 4 00000800:00000200:13.0:1547251680.186812:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:13.0:1547251680.186813:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000100:00000200:0.0:1547251680.186814:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447872, offset 0 00000800:00000200:13.0:1547251680.186815:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:13.0:1547251680.186817:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37937800 x1622384012447856/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251687 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251680.186818:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:13.0:1547251680.186821:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990990 00000400:00000200:13.0:1547251680.186822:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:0.0:1547251680.186822:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:13.0:1547251680.186824:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:0.0:1547251680.186824:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23096] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:13.0:1547251680.186826:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:13.0:1547251680.186827:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000400:00000200:0.0:1547251680.186827:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23096] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23096] 00000400:00000200:0.0:1547251680.186830:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:0.0:1547251680.186831:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251680.186833:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251680.186834:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9771, 9772] 00000400:00000200:0.0:1547251680.186836:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:0.0:1547251680.186840:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251680.186843:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:0.0:1547251680.186846:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251680.186849:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:0.0:1547251680.186850:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:0.0:1547251680.186853:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:7.2:1547251680.186868:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:13.0:1547251680.186880:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:13.0:1547251680.186886:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:13.0:1547251680.186888:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c1e00 x1622384012447872/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251687 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251680.186892:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665550 00000400:00000200:13.0:1547251680.186893:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:15.0:1547251680.186894:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:13.0:1547251680.186895:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251680.186897:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:9.2:1547251680.186931:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.2:1547251680.186938:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:9.2:1547251680.186942:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:15.0:1547251680.186953:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:13.0:1547251680.186956:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:15.0:1547251680.186957:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.93@o2ib10 00000800:00000200:13.0:1547251680.186959:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251680.186961:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251680.186962:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251680.186965:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05070 00000400:00000200:13.0:1547251680.186966:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05060 00000400:00000200:15.0:1547251680.186968:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x8006d [1] + 192 00000400:00000200:13.0:1547251680.186971:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x8005d [1] + 192 00000100:00000200:15.0:1547251680.186976:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37937800 x1622384012447856/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251687 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251680.186987:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:13.0:1547251680.186988:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07e900 x1622384012447840/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251687 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251680.186989:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:15.0:1547251680.186991:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:15.0:1547251680.186992:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:15.0:1547251680.186994:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:15.0:1547251680.186995:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000400:00000200:13.0:1547251680.186996:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:15.0:1547251680.186998:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:13.0:1547251680.186998:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000100:00000200:6.0:1547251680.186998:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37937800 x1622384012447856/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251687 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251680.187000:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000400:00000200:15.0:1547251680.187001:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05080 00000800:00000200:13.0:1547251680.187001:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:13.0:1547251680.187003:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:13.0:1547251680.187004:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000100:00000200:6.0:1547251680.187004:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37937800 x1622384012447856/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251687 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251680.187005:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:6.0:1547251680.187010:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990d48 00000400:00000200:15.0:1547251680.187012:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x8007d [1] + 192 00010000:00080000:6.0:1547251680.187013:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ae37937800 x1622384012447856/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251687 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:15.0:1547251680.187015:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c1e00 x1622384012447872/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251687 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251680.187022:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251680.187024:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:15.0:1547251680.187025:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:15.0:1547251680.187025:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000100:00000200:0.0:1547251680.187033:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c1e00 x1622384012447872/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251687 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251680.187038:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c1e00 x1622384012447872/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251687 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251680.187042:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00010000:00080000:0.0:1547251680.187046:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c1e00 x1622384012447872/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251687 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:11.0:1547251680.187047:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07e900 x1622384012447840/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251687 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251680.187053:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07e900 x1622384012447840/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251687 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251680.187058:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b440 00010000:00080000:11.0:1547251680.187061:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b17b07e900 x1622384012447840/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251687 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:10.0:1547251680.307689:0:12658:0:(o2iblnd_cb.c:1409:kiblnd_connect_peer()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)++ 00000800:00000200:10.0:1547251680.307704:0:12658:0:(o2iblnd_cb.c:1362:kiblnd_resolve_addr()) bound to port 1023 00000800:00000200:10.0:1547251680.307705:0:12658:0:(o2iblnd_cb.c:3557:kiblnd_connd()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (3)-- 00000800:00000400:10.0:1547251680.307712:0:12658:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.2.10.184@o2ib10: 0 seconds 00000800:00000200:9.0:1547251680.307713:0:16171:0:(o2iblnd_cb.c:3156:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Addr resolved: 0 00000400:00000200:10.0:1547251680.307721:0:12658:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = -110, hstatus = 6 00000800:00000200:9.0:1547251680.307721:0:16171:0:(o2iblnd_cb.c:3173:kiblnd_cm_callback()) 10.2.10.184@o2ib10: connection bound to ib1:10.1.11.90:mlx5_1 00000400:00000200:10.0:1547251680.307724:0:12658:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.2.10.184@o2ib10: PUT: LOCAL_TIMEOUT 00000400:00000100:10.0:1547251680.307726:0:12658:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.2.10.184@o2ib10 exceeded retry count 0 00000400:00000200:10.0:1547251680.307729:0:12658:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06aaa0 00000800:00000200:9.0:1547251680.308249:0:16171:0:(o2iblnd_cb.c:3197:kiblnd_cm_callback()) 10.2.10.184@o2ib10 Route resolved: 0 00000800:00000200:9.0:1547251680.308256:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 0: ffff97b17b3ef000 0xd3b3ef000(0xd3b3ef000) 00000800:00000200:9.0:1547251680.308257:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 1: ffff97b17b3ee000 0xd3b3ee000(0xd3b3ee000) 00000800:00000200:9.0:1547251680.308258:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 2: ffff97ac1b2d1000 0x7db2d1000(0x7db2d1000) 00000800:00000200:9.0:1547251680.308259:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 3: ffff97ac1b2d0000 0x7db2d0000(0x7db2d0000) 00000800:00000200:9.0:1547251680.308260:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 4: ffff97b17b0bd000 0xd3b0bd000(0xd3b0bd000) 00000800:00000200:9.0:1547251680.308261:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 5: ffff97b17b0bc000 0xd3b0bc000(0xd3b0bc000) 00000800:00000200:9.0:1547251680.308262:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 6: ffff97a744b59000 0x304b59000(0x304b59000) 00000800:00000200:9.0:1547251680.308263:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 7: ffff97a744b58000 0x304b58000(0x304b58000) 00000800:00000200:9.0:1547251680.308264:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 8: ffff97b17b2e5000 0xd3b2e5000(0xd3b2e5000) 00000800:00000200:9.0:1547251680.308265:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 9: ffff97b17b2e4000 0xd3b2e4000(0xd3b2e4000) 00000800:00000200:9.0:1547251680.308265:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 10: ffff97b81d5db000 0x13dd5db000(0x13dd5db000) 00000800:00000200:9.0:1547251680.308266:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 11: ffff97b81d5da000 0x13dd5da000(0x13dd5da000) 00000800:00000200:9.0:1547251680.308267:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 12: ffff97ba8774f000 0x164774f000(0x164774f000) 00000800:00000200:9.0:1547251680.308268:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 13: ffff97ba8774e000 0x164774e000(0x164774e000) 00000800:00000200:9.0:1547251680.308269:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 14: ffff97b4e7edb000 0x10a7edb000(0x10a7edb000) 00000800:00000200:9.0:1547251680.308270:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 15: ffff97b4e7eda000 0x10a7eda000(0x10a7eda000) 00000800:00000200:9.0:1547251680.308271:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 16: ffff97ba91ae1000 0x1651ae1000(0x1651ae1000) 00000800:00000200:9.0:1547251680.308271:0:16171:0:(o2iblnd.c:1340:kiblnd_map_rx_descs()) rx 17: ffff97ae6a9fe000 0xa2a9fe000(0xa2a9fe000) 00000800:00000200:9.0:1547251680.308883:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308885:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308886:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308886:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308888:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308888:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308889:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308890:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308890:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308891:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308891:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308892:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308893:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308893:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308894:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308895:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308895:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308896:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308896:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308897:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308898:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308898:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308899:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308899:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308900:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308901:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308901:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308902:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308902:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308903:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308904:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308904:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308905:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308905:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:9.0:1547251680.308906:0:16171:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:9.0:1547251680.308907:0:16171:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba8baea400] (20)-- 00000800:00000100:12.0:1547251680.309229:0:70:0:(o2iblnd_cb.c:2822:kiblnd_check_reconnect()) 10.2.10.184@o2ib10: don't reconnect (no need), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:12.0:1547251680.309232:0:70:0:(o2iblnd_cb.c:2848:kiblnd_rejected()) 10.2.10.184@o2ib10 rejected: no listener at 987 00000800:00000200:12.0:1547251680.309235:0:70:0:(o2iblnd_cb.c:2281:kiblnd_connreq_done()) 10.2.10.184@o2ib10: active(1), version(12), status(-111) 00000800:00000200:12.0:1547251680.309237:0:70:0:(o2iblnd.c:417:kiblnd_unlink_peer_locked()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (2)-- 00000400:00000200:12.0:1547251680.309241:0:70:0:(router.c:1756:lnet_notify()) 10.1.11.90@o2ib10 notifying 10.2.10.184@o2ib10: down 00000400:00000200:12.0:1547251680.309242:0:70:0:(router.c:120:lnet_notify_locked()) Old news 00000800:00000200:8.2:1547251680.309277:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba8baea400] (19)++ 00000800:00000200:13.0:1547251680.309300:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (20)++ 00000800:00000200:13.0:1547251680.309308:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (21)-- 00000800:00000200:13.0:1547251680.309309:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (20)-- 00000800:00000200:13.0:1547251680.309311:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (19)-- 00000800:00000200:12.0:1547251680.309314:0:70:0:(o2iblnd_cb.c:3247:kiblnd_cm_callback()) conn[ffff97ba8baea400] (18)-- 00000800:00000200:8.2:1547251680.309415:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba8baea400] (17)++ 00000800:00000200:13.0:1547251680.309427:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (18)++ 00000800:00000200:13.0:1547251680.309432:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (19)-- 00000800:00000200:13.0:1547251680.309433:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (18)-- 00000800:00000200:13.0:1547251680.309434:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (17)++ 00000800:00000200:13.0:1547251680.309435:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (18)-- 00000800:00000200:13.0:1547251680.309435:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (17)-- 00000800:00000200:13.0:1547251680.309436:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (16)++ 00000800:00000200:13.0:1547251680.309437:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (17)-- 00000800:00000200:13.0:1547251680.309438:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (16)-- 00000800:00000200:13.0:1547251680.309439:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (15)++ 00000800:00000200:13.0:1547251680.309442:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (16)-- 00000800:00000200:13.0:1547251680.309442:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (15)-- 00000800:00000200:13.0:1547251680.309443:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (14)++ 00000800:00000200:13.0:1547251680.309444:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (15)-- 00000800:00000200:13.0:1547251680.309445:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (14)-- 00000800:00000200:15.0:1547251680.309446:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (13)++ 00000800:00000200:15.0:1547251680.309453:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (14)-- 00000800:00000200:15.0:1547251680.309454:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (13)-- 00000800:00000200:15.0:1547251680.309455:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (12)++ 00000800:00000200:15.0:1547251680.309458:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (13)-- 00000800:00000200:15.0:1547251680.309459:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (12)-- 00000800:00000200:15.0:1547251680.309460:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (11)++ 00000800:00000200:15.0:1547251680.309461:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (12)-- 00000800:00000200:13.0:1547251680.309461:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (12)++ 00000800:00000200:15.0:1547251680.309462:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (12)-- 00000800:00000200:15.0:1547251680.309463:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (11)++ 00000800:00000200:13.0:1547251680.309463:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (11)-- 00000800:00000200:13.0:1547251680.309464:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (11)-- 00000800:00000200:15.0:1547251680.309465:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (10)-- 00000800:00000200:13.0:1547251680.309465:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (10)++ 00000800:00000200:15.0:1547251680.309466:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (10)-- 00000800:00000200:15.0:1547251680.309468:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (9)++ 00000800:00000200:13.0:1547251680.309468:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (9)-- 00000800:00000200:13.0:1547251680.309469:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (9)-- 00000800:00000200:15.0:1547251680.309470:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (8)-- 00000800:00000200:13.0:1547251680.309470:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (8)++ 00000800:00000200:15.0:1547251680.309471:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (8)-- 00000800:00000200:15.0:1547251680.309472:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (7)++ 00000800:00000200:13.0:1547251680.309472:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (7)-- 00000800:00000200:13.0:1547251680.309473:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (7)-- 00000800:00000200:15.0:1547251680.309474:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (6)-- 00000800:00000200:13.0:1547251680.309474:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (6)++ 00000800:00000200:15.0:1547251680.309475:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (6)-- 00000800:00000200:15.0:1547251680.309476:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (5)++ 00000800:00000200:13.0:1547251680.309476:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (5)-- 00000800:00000200:13.0:1547251680.309477:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (5)-- 00000800:00000200:15.0:1547251680.309478:0:12666:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (4)-- 00000800:00000200:13.0:1547251680.309479:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba8baea400] (4)++ 00000800:00000200:15.0:1547251680.309481:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (4)-- 00000800:00000200:13.0:1547251680.309481:0:12665:0:(o2iblnd_cb.c:158:kiblnd_drop_rx()) conn[ffff97ba8baea400] (3)-- 00000800:00000200:15.0:1547251680.309482:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (3)-- 00000800:00000200:13.0:1547251680.309483:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba8baea400] (1)-- 00000800:00000200:9.0:1547251680.310227:0:12658:0:(o2iblnd.c:1056:kiblnd_destroy_conn()) peer_ni[ffff97badfad0800] -> 10.2.10.184@o2ib10 (1)-- 00000100:00000200:4.0:1547251681.786686:0:12676:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012447920, portal 10 00000100:00000200:4.0:1547251681.786689:0:12676:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012447920, offset 0 00000100:00000200:5.0:1547251681.786693:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012447888, portal 25 00000400:00000200:4.0:1547251681.786693:0:12676:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:5.0:1547251681.786695:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 26, xid 1622384012447888, offset 0 00000400:00000200:4.0:1547251681.786696:0:12676:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251681.786698:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251681.786700:0:12676:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c050b0 00000400:00000200:5.0:1547251681.786701:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251681.786704:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 26 MB=0x5c38cb6c05090 00000400:00000200:4.0:1547251681.786704:0:12676:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 65184 00000400:00000200:5.0:1547251681.786706:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 1a from 12345-0@lo of length 224/224 into md 0xd5679 [1] + 0 00000100:00000200:4.0:1547251681.786707:0:12676:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:5.0:1547251681.786709:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b17b06a220 00000100:00000200:5.0:1547251681.786711:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service mgs 00000400:00000200:4.0:1547251681.786712:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251681.786715:0:12676:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c435a00 x1622384012447920/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251681.786719:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b06a220 00000400:00000200:5.0:1547251681.786720:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251681.786721:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012447936, portal 10 00000100:00000200:6.0:1547251681.786721:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012447904, portal 10 00000400:00000200:4.0:1547251681.786722:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b110 00000100:00000200:5.0:1547251681.786723:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c437800 x1622384012447888/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251688 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251681.786723:0:12676:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251681.786724:0:12676:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:5.0:1547251681.786729:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ade58 00000400:00000200:5.0:1547251681.786730:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:7.0:1547251681.786731:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012447936, offset 0 00000100:00000200:6.0:1547251681.786731:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012447904, offset 0 00000400:00000200:5.0:1547251681.786732:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251681.786734:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:5.0:1547251681.786736:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1622384012447952, portal 10 00000400:00000200:7.0:1547251681.786737:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251681.786738:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251681.786739:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:5.0:1547251681.786739:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1622384012447952, offset 0 00000400:00000200:5.0:1547251681.786741:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251681.786742:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c050c0 00000400:00000200:6.0:1547251681.786742:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c050a0 00000400:00000200:5.0:1547251681.786744:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251681.786745:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 65408 00000100:00000200:4.0:1547251681.786746:0:13355:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447920 00000400:00000200:5.0:1547251681.786747:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 12 MB=0x5c38cb6c050d0 00000400:00000200:7.0:1547251681.786748:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 65632 00000100:00000200:6.0:1547251681.786751:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00010000:00000200:4.0:1547251681.786751:0:13355:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97b82444d100 x1622384012447920/t0(0) o400->cache1-MDT0000-lwp-OST0000_UUID@0@lo:742/0 lens 224/224 e 0 to 0 dl 1547251687 ref 1 fl Interpret:H/0/0 rc 0/0 00000400:00000200:6.0:1547251681.786758:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251681.786759:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index c from 12345-0@lo of length 224/224 into md 0x515 [1] + 65856 00000100:00000200:7.0:1547251681.786762:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:7.0:1547251681.786766:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251681.786767:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c437200 x1622384012447904/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251681.786774:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459907f8 00000400:00000200:6.0:1547251681.786775:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:5.0:1547251681.786775:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service mdt 00000400:00000200:6.0:1547251681.786776:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:2.0:1547251681.786776:0:12929:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447888 00000400:00000200:5.0:1547251681.786780:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251681.786780:0:13355:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012447920, offset 192 00000100:00000200:7.0:1547251681.786781:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c433600 x1622384012447936/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00010000:00000200:2.0:1547251681.786781:0:12929:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ba7349bc50 x1622384012447888/t0(0) o400->a65be517-33e7-fa36-0de3-da9e02df4f8d@0@lo:742/0 lens 224/224 e 0 to 0 dl 1547251687 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:7.0:1547251681.786787:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eaa18 00000400:00000200:7.0:1547251681.786788:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:2.0:1547251681.786788:0:12929:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 25, xid 1622384012447888, offset 192 00000100:00000200:5.0:1547251681.786789:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c431b00 x1622384012447952/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251681.786790:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:2.0:1547251681.786791:0:12929:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:2.0:1547251681.786794:0:12929:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251681.786795:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad770 00000400:00000200:5.0:1547251681.786796:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:2.0:1547251681.786796:0:12929:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 25 MB=0x5c38cb6c05090 00000400:00000200:5.0:1547251681.786797:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251681.786797:0:13355:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:7.0:1547251681.786799:0:13324:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447904 00000400:00000200:4.0:1547251681.786799:0:13355:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:2.0:1547251681.786799:0:12929:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 19 from 12345-0@lo of length 192/192 into md 0xaf5f1 [1] + 192 00000100:00000200:6.0:1547251681.786801:0:13338:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447936 00000400:00000200:4.0:1547251681.786802:0:13355:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c050b0 00000100:00000200:2.0:1547251681.786802:0:12929:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c437800 x1622384012447888/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00010000:00000200:7.0:1547251681.786805:0:13324:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97a765df6c00 x1622384012447904/t0(0) o400->cache1-MDT0000-lwp-MDT0000_UUID@0@lo:742/0 lens 224/224 e 0 to 0 dl 1547251687 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000200:6.0:1547251681.786805:0:13338:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae37931b00 x1622384012447936/t0(0) o400->cache1-MDT0000-lwp-OST0002_UUID@0@lo:742/0 lens 224/224 e 0 to 0 dl 1547251687 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:5.0:1547251681.786808:0:12980:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012447952 00000400:00000200:2.0:1547251681.786808:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251681.786809:0:13355:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaf5e1 [1] + 192 00010000:00000200:5.0:1547251681.786811:0:12980:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97ae5c436f00 x1622384012447952/t0(0) o400->cache1-MDT0000-lwp-OST0001_UUID@0@lo:742/0 lens 224/224 e 0 to 0 dl 1547251687 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00000200:7.0:1547251681.786812:0:13324:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012447904, offset 192 00000100:00000200:6.0:1547251681.786812:0:13338:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012447936, offset 192 00000400:00000200:2.0:1547251681.786812:0:12929:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b630fe9330 00000400:00000200:2.0:1547251681.786813:0:12929:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:4.0:1547251681.786814:0:13355:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c435a00 x1622384012447920/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251681.786814:0:12929:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251681.786815:0:13324:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251681.786816:0:13338:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251681.786818:0:13324:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:5.0:1547251681.786818:0:12980:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 10, xid 1622384012447952, offset 192 00000100:00000200:2.0:1547251681.786818:0:12929:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 26 00000400:00000200:6.0:1547251681.786819:0:13338:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251681.786821:0:13324:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c050a0 00000400:00000200:6.0:1547251681.786821:0:13338:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c050c0 00000400:00000200:5.0:1547251681.786821:0:12980:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251681.786821:0:13355:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251681.786823:0:13355:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b110 00000400:00000200:7.0:1547251681.786824:0:13324:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaf601 [1] + 192 00000400:00000200:5.0:1547251681.786824:0:12980:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251681.786824:0:13355:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251681.786825:0:13355:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:6.0:1547251681.786827:0:13338:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaf611 [1] + 192 00000400:00000200:5.0:1547251681.786827:0:12980:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 192 into portal 10 MB=0x5c38cb6c050d0 00000100:00000200:7.0:1547251681.786830:0:13324:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c437200 x1622384012447904/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251681.786837:0:13324:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251681.786838:0:12980:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index a from 12345-0@lo of length 192/192 into md 0xaf621 [1] + 192 00000400:00000200:7.0:1547251681.786839:0:13324:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eaa18 00000400:00000200:7.0:1547251681.786840:0:13324:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251681.786841:0:13324:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251681.786842:0:13338:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c433600 x1622384012447936/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251681.786849:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251681.786850:0:12676:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c435a00 x1622384012447920/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251681.786851:0:13338:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459907f8 00000400:00000200:6.0:1547251681.786852:0:13338:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251681.786853:0:13338:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251681.786855:0:12676:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c435a00 x1622384012447920/t0(0) o400->cache1-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251681.786860:0:12676:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040be58 00000100:00000200:5.0:1547251681.786862:0:12980:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c431b00 x1622384012447952/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251681.786867:0:12980:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:7.0:1547251681.786869:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c433600 x1622384012447936/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251681.786869:0:12980:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad770 00000400:00000200:5.0:1547251681.786870:0:12980:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251681.786872:0:12980:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251681.786874:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c433600 x1622384012447936/t0(0) o400->cache1-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251681.786879:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea880 00000100:00000200:6.0:1547251681.786881:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c437200 x1622384012447904/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251681.786888:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c437200 x1622384012447904/t0(0) o400->cache1-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251681.786894:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990d48 00000100:00000200:5.0:1547251681.786896:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c437800 x1622384012447888/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251681.786901:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c437800 x1622384012447888/t0(0) o400->MGC10.0.11.90@o2ib10@0@lo:26/25 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251681.786904:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad330 00000100:00000200:5.0:1547251681.786907:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c431b00 x1622384012447952/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251681.786911:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c431b00 x1622384012447952/t0(0) o400->cache1-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 0 dl 1547251688 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251681.786915:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ade58 00000100:00000200:8.0:1547251682.298682:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447968, portal 4 00000100:00000200:8.0:1547251682.298684:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447968, offset 0 00000400:00000200:8.0:1547251682.298687:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251682.298690:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251682.298693:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23097] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251682.298696:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23096] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23097] 00000100:00000200:13.0:1547251682.298698:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012447984, portal 4 00000400:00000200:8.0:1547251682.298699:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000100:00000200:13.0:1547251682.298701:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012447984, offset 0 00000400:00000200:8.0:1547251682.298701:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251682.298702:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251682.298703:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9777, 9776] 00000400:00000200:13.0:1547251682.298704:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:8.0:1547251682.298705:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:13.0:1547251682.298707:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:13.0:1547251682.298710:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23097] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251682.298710:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000400:00000200:13.0:1547251682.298713:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23097] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23097] 00000800:00000200:8.0:1547251682.298713:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:8.0:1547251682.298715:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000400:00000200:13.0:1547251682.298716:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:13.0:1547251682.298717:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251682.298717:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:8.0:1547251682.298718:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:13.0:1547251682.298719:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251682.298720:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9777, 9777] 00000800:00000200:8.0:1547251682.298721:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (22)-- 00000400:00000200:13.0:1547251682.298722:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:13.0:1547251682.298726:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251682.298729:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:13.0:1547251682.298732:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0400] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251682.298734:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57202400] (20)++ 00000800:00000200:13.0:1547251682.298735:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (21)++ 00000800:00000200:13.0:1547251682.298737:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57202400] (22)-- 00000100:00000200:2.0:1547251682.298737:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448000, portal 4 00000100:00000200:2.0:1547251682.298740:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448000, offset 0 00000400:00000200:2.0:1547251682.298743:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:2.0:1547251682.298746:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:2.0:1547251682.298748:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23098] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251682.298751:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23097] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23098] 00000400:00000200:2.0:1547251682.298753:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251682.298756:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251682.298757:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251682.298759:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9778, 9777] 00000400:00000200:2.0:1547251682.298761:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:2.0:1547251682.298765:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251682.298768:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:2.0:1547251682.298771:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4900] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251682.298772:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:2.0:1547251682.298773:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:7.2:1547251682.298775:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:2.0:1547251682.298776:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:5.2:1547251682.298792:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00000800:00000200:10.0:1547251682.298803:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:10.0:1547251682.298807:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (24)-- 00000800:00000200:11.0:1547251682.298808:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251682.298811:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000100:00000200:10.0:1547251682.298812:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824756900 x1622384012447968/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:11.0:1547251682.298816:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af0300 x1622384012447984/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251682.298819:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688f68 00000400:00000200:10.0:1547251682.298821:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251682.298822:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b3b8 00000400:00000200:11.0:1547251682.298823:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251682.298824:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000400:00000200:11.0:1547251682.298827:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251682.298827:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251682.298829:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:11.0:1547251682.298830:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:10.0:1547251682.298830:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:11.0:1547251682.298831:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:11.0:1547251682.298833:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000100:00000200:10.0:1547251682.298833:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f3900 x1622384012448000/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251682.298838:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b630fe9440 00000400:00000200:10.0:1547251682.298840:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251682.298843:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251682.298845:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:5.2:1547251682.298891:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251682.298897:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:11.0:1547251682.298904:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251682.298908:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000800:00000200:10.0:1547251682.298909:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251682.298912:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251682.298913:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251682.298916:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251682.298918:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c050f0 00000400:00000200:10.0:1547251682.298920:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c050e0 00000400:00000200:11.0:1547251682.298922:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7739 [1] + 192 00000400:00000200:10.0:1547251682.298928:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7719 [1] + 192 00000100:00000200:11.0:1547251682.298931:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af0300 x1622384012447984/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251682.298952:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251682.298954:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000100:00000200:10.0:1547251682.298954:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824756900 x1622384012447968/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251682.298956:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251682.298957:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251682.298959:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000800:00000200:11.0:1547251682.298961:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:11.0:1547251682.298963:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251682.298967:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251682.298967:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:13.0:1547251682.298969:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af0300 x1622384012447984/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251682.298969:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:10.0:1547251682.298970:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (24)-- 00000400:00000200:11.0:1547251682.298971:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05100 00000800:00000200:10.0:1547251682.298971:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251682.298973:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000100:00000200:13.0:1547251682.298975:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af0300 x1622384012447984/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251682.298979:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b990 00000400:00000200:11.0:1547251682.298982:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7749 [1] + 192 00010000:00080000:13.0:1547251682.298983:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af0300 x1622384012447984/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251682.298987:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824756900 x1622384012447968/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251682.298993:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824756900 x1622384012447968/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251682.298997:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688aa0 00000100:00000200:11.0:1547251682.298999:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f3900 x1622384012448000/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251682.299000:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824756900 x1622384012447968/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:11.0:1547251682.299007:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251682.299008:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:11.0:1547251682.299009:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251682.299010:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:2.0:1547251682.299016:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f3900 x1622384012448000/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251682.299022:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f3900 x1622384012448000/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251682.299026:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b630fe93b8 00010000:00080000:2.0:1547251682.299030:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f3900 x1622384012448000/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251682.762681:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448016, portal 4 00000100:00000200:8.0:1547251682.762683:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448016, offset 0 00000400:00000200:8.0:1547251682.762686:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251682.762689:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251682.762691:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23098] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:13.0:1547251682.762694:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448032, portal 4 00000400:00000200:8.0:1547251682.762694:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23098] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23098] 00000100:00000200:13.0:1547251682.762697:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448032, offset 0 00000400:00000200:8.0:1547251682.762697:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251682.762698:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251682.762700:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251682.762701:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251682.762701:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9770, 9770] 00000400:00000200:8.0:1547251682.762703:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:13.0:1547251682.762704:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:13.0:1547251682.762706:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23099] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251682.762708:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:13.0:1547251682.762711:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23098] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23099] 00000400:00000200:13.0:1547251682.762713:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:8.0:1547251682.762713:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:13.0:1547251682.762715:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:13.0:1547251682.762716:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:8.0:1547251682.762716:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:13.0:1547251682.762717:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9770, 9771] 00000800:00000200:8.0:1547251682.762717:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:8.0:1547251682.762718:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000400:00000200:13.0:1547251682.762719:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:8.0:1547251682.762721:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000400:00000200:13.0:1547251682.762724:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:13.0:1547251682.762727:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000100:00000200:0.0:1547251682.762729:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448048, portal 4 00000800:00000200:13.0:1547251682.762731:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:13.0:1547251682.762733:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000100:00000200:0.0:1547251682.762733:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448048, offset 0 00000800:00000200:13.0:1547251682.762735:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:0.0:1547251682.762737:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:13.0:1547251682.762739:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:12.2:1547251682.762739:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000400:00000200:0.0:1547251682.762740:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:0.0:1547251682.762743:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23099] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251682.762746:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23099] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23099] 00000800:00000200:0.2:1547251682.762749:0:12673:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:0.0:1547251682.762766:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:0.0:1547251682.762768:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251682.762769:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251682.762770:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9771, 9771] 00000400:00000200:0.0:1547251682.762772:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:0.0:1547251682.762777:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251682.762780:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:0.0:1547251682.762783:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251682.762785:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (22)++ 00000800:00000200:0.0:1547251682.762786:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (23)++ 00000800:00000200:0.0:1547251682.762788:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (24)-- 00000800:00000200:5.0:1547251682.762798:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (23)++ 00000800:00000200:5.0:1547251682.762802:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (24)-- 00000800:00000200:4.0:1547251682.762802:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251682.762805:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:5.0:1547251682.762807:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824754200 x1622384012448016/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251682.762808:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af4e00 x1622384012448032/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251682.762814:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688660 00000400:00000200:4.0:1547251682.762814:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792bd48 00000400:00000200:5.0:1547251682.762816:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251682.762816:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251682.762819:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251682.762819:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251682.762822:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251682.762823:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251682.762824:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251682.762826:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:5.0:1547251682.762827:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251682.762828:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:5.0:1547251682.762830:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c1b00 x1622384012448048/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251682.762836:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665440 00000400:00000200:5.0:1547251682.762837:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251682.762840:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:5.0:1547251682.762841:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:12.2:1547251682.762902:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:0.2:1547251682.762908:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:0.2:1547251682.762913:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:4.0:1547251682.762916:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251682.762920:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.92@o2ib10 00000800:00000200:5.0:1547251682.762921:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251682.762924:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251682.762925:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251682.762928:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251682.762930:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05110 00000400:00000200:5.0:1547251682.762932:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05120 00000400:00000200:4.0:1547251682.762934:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4695 [1] + 192 00000400:00000200:5.0:1547251682.762939:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe46a5 [1] + 192 00000100:00000200:4.0:1547251682.762952:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824754200 x1622384012448016/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251682.762965:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251682.762968:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af4e00 x1622384012448032/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251682.762968:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251682.762970:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251682.762971:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251682.762973:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251682.762975:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251682.762978:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251682.762979:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000100:00000200:8.0:1547251682.762980:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824754200 x1622384012448016/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251682.762980:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251682.762982:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251682.762983:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000400:00000200:4.0:1547251682.762983:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05130 00000800:00000200:5.0:1547251682.762985:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:8.0:1547251682.762986:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824754200 x1622384012448016/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251682.762987:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:5.0:1547251682.762988:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:8.0:1547251682.762990:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688aa0 00000400:00000200:4.0:1547251682.762993:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe46b5 [1] + 192 00010000:00080000:8.0:1547251682.762994:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824754200 x1622384012448016/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:13.0:1547251682.762999:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af4e00 x1622384012448032/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:13.0:1547251682.763004:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af4e00 x1622384012448032/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251682.763008:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae3792b990 00000100:00000200:4.0:1547251682.763011:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c1b00 x1622384012448048/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:13.0:1547251682.763012:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af4e00 x1622384012448032/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:4.0:1547251682.763019:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251682.763020:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251682.763021:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:4.0:1547251682.763022:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000100:00000200:0.0:1547251682.763029:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c1b00 x1622384012448048/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251682.763034:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c1b00 x1622384012448048/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251682.763045:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00010000:00080000:0.0:1547251682.763048:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c1b00 x1622384012448048/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251689 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251683.162673:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448064, portal 4 00000100:00000200:8.0:1547251683.162676:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448064, offset 0 00000400:00000200:8.0:1547251683.162678:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251683.162681:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251683.162684:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c05140 00000400:00000200:8.0:1547251683.162686:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4e19 [1] + 0 00000400:00000200:8.0:1547251683.162688:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b81d688880 00000100:00000200:8.0:1547251683.162690:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251683.162698:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688880 00000400:00000200:8.0:1547251683.162700:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251683.162704:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824757800 x1622384012448064/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251690 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251683.162710:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688dd0 00000400:00000200:8.0:1547251683.162711:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251683.162713:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251683.162720:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012448064 00010000:00000200:9.0:1547251683.162730:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c2dfc50 x1622384012448064/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:744/0 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:9.0:1547251683.162737:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012448064, offset 192 00000400:00000200:9.0:1547251683.162740:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251683.162743:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251683.162745:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c05140 00000400:00000200:9.0:1547251683.162747:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf661 [1] + 192 00000100:00000200:5.0:1547251683.162747:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448080, portal 4 00000100:00000200:9.0:1547251683.162750:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824757800 x1622384012448064/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251690 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251683.162750:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448080, offset 0 00000400:00000200:5.0:1547251683.162752:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251683.162755:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251683.162758:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251683.162758:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c05150 00000400:00000200:5.0:1547251683.162760:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2e95 [1] + 0 00000400:00000200:9.0:1547251683.162761:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea088 00000400:00000200:9.0:1547251683.162761:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251683.162763:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040b7f8 00000400:00000200:9.0:1547251683.162764:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251683.162764:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:8.0:1547251683.162768:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824757800 x1622384012448064/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251690 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251683.162770:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:5.0:1547251683.162772:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b7f8 00000400:00000200:5.0:1547251683.162773:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251683.162774:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824757800 x1622384012448064/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251690 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251683.162776:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37937b00 x1622384012448080/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251690 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251683.162780:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688aa0 00010000:00080000:8.0:1547251683.162785:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824757800 x1622384012448064/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251690 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251683.162791:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad330 00000400:00000200:5.0:1547251683.162792:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251683.162794:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251683.162808:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012448080 00010000:00000200:4.0:1547251683.162818:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11d450 x1622384012448080/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:744/0 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251683.162825:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012448080, offset 192 00000400:00000200:4.0:1547251683.162827:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251683.162830:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251683.162832:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c05150 00000400:00000200:4.0:1547251683.162835:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf671 [1] + 192 00000100:00000200:4.0:1547251683.162837:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37937b00 x1622384012448080/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251690 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251683.162846:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:4.0:1547251683.162848:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040be58 00000400:00000200:4.0:1547251683.162849:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251683.162852:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251683.162857:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37937b00 x1622384012448080/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251690 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251683.162858:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251683.162863:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37937b00 x1622384012448080/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251690 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251683.162868:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ade58 00010000:00080000:5.0:1547251683.162872:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37937b00 x1622384012448080/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251690 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251683.178701:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448096, portal 4 00000100:00000200:6.0:1547251683.178704:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448096, offset 0 00000400:00000200:6.0:1547251683.178706:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251683.178709:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251683.178712:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c05160 00000400:00000200:6.0:1547251683.178714:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2ec5 [1] + 0 00000400:00000200:6.0:1547251683.178717:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040baa0 00000100:00000200:6.0:1547251683.178719:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:6.0:1547251683.178726:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040baa0 00000400:00000200:6.0:1547251683.178727:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:6.0:1547251683.178729:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37933000 x1622384012448096/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251690 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251683.178734:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459907f8 00000400:00000200:6.0:1547251683.178734:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251683.178736:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251683.178757:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012448096 00010000:00000200:5.0:1547251683.178766:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11d050 x1622384012448096/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:744/0 lens 224/368 e 0 to 0 dl 1547251689 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:5.0:1547251683.178773:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012448096, offset 192 00000400:00000200:5.0:1547251683.178775:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251683.178778:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251683.178781:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c05160 00000400:00000200:5.0:1547251683.178783:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf681 [1] + 192 00000100:00000200:5.0:1547251683.178786:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37933000 x1622384012448096/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251690 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251683.178796:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251683.178798:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ade58 00000400:00000200:5.0:1547251683.178799:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251683.178801:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251683.178804:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37933000 x1622384012448096/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251690 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251683.178806:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:6.0:1547251683.178808:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37933000 x1622384012448096/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251690 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251683.178812:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990d48 00010000:00080000:6.0:1547251683.178815:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37933000 x1622384012448096/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251690 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:1.0:1547251685.194676:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448128, portal 4 00000100:00000200:1.0:1547251685.194679:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448128, offset 0 00000400:00000200:1.0:1547251685.194682:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000100:00000200:10.0:1547251685.194683:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448112, portal 4 00000400:00000200:1.0:1547251685.194685:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000100:00000200:10.0:1547251685.194686:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448112, offset 0 00000400:00000200:1.0:1547251685.194687:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23100] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:10.0:1547251685.194689:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:1.0:1547251685.194690:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23099] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23100] 00000400:00000200:10.0:1547251685.194692:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:1.0:1547251685.194693:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:1.0:1547251685.194694:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251685.194695:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23100] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251685.194696:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251685.194697:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9772, 9772] 00000400:00000200:10.0:1547251685.194698:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23100] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23100] 00000400:00000200:1.0:1547251685.194699:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:10.0:1547251685.194700:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:10.0:1547251685.194702:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251685.194704:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251685.194704:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000400:00000200:10.0:1547251685.194705:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 7], s:[9772, 9773] 00000800:00000200:1.0:1547251685.194706:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:10.0:1547251685.194707:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000800:00000200:1.0:1547251685.194709:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251685.194711:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000400:00000200:10.0:1547251685.194712:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:1.0:1547251685.194712:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:1.0:1547251685.194714:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:10.0:1547251685.194715:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:10.0:1547251685.194717:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:10.0:1547251685.194719:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:10.0:1547251685.194720:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:10.0:1547251685.194722:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:9.2:1547251685.194734:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:7.2:1547251685.194743:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:4.0:1547251685.194755:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448144, portal 4 00000100:00000200:4.0:1547251685.194759:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448144, offset 0 00000400:00000200:4.0:1547251685.194764:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:4.0:1547251685.194767:0:12677:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:4.0:1547251685.194770:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23101] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:4.0:1547251685.194773:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23100] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23101] 00000400:00000200:4.0:1547251685.194775:0:12677:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:4.0:1547251685.194776:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251685.194778:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251685.194779:0:12677:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9773, 9773] 00000400:00000200:4.0:1547251685.194781:0:12677:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:4.0:1547251685.194786:0:12677:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251685.194788:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:4.0:1547251685.194788:0:12677:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:4.0:1547251685.194791:0:12677:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251685.194792:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:4.0:1547251685.194793:0:12677:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:4.0:1547251685.194794:0:12677:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (23)++ 00000100:00000200:15.0:1547251685.194795:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b6300 x1622384012448128/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251692 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251685.194797:0:12677:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (24)-- 00000400:00000200:15.0:1547251685.194800:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2000 00000400:00000200:15.0:1547251685.194801:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:15.0:1547251685.194804:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251685.194806:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:15.0:1547251685.194808:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:13.0:1547251685.194809:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:15.0:1547251685.194810:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:15.0:1547251685.194811:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:15.0:1547251685.194813:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df4500 x1622384012448144/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251692 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:13.0:1547251685.194813:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960eaa00 x1622384012448112/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251692 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251685.194817:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b000 00000400:00000200:15.0:1547251685.194818:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251685.194818:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914a18 00000400:00000200:13.0:1547251685.194819:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:15.0:1547251685.194820:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:15.0:1547251685.194821:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:15.0:1547251685.194822:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:13.0:1547251685.194822:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:15.0:1547251685.194824:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:13.0:1547251685.194824:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:9.2:1547251685.194887:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:7.2:1547251685.194892:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:15.0:1547251685.194899:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:15.0:1547251685.194902:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.0.11.93@o2ib10 00000800:00000200:13.0:1547251685.194905:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:15.0:1547251685.194906:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:7.2:1547251685.194906:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:13.0:1547251685.194907:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251685.194909:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05180 00000400:00000200:13.0:1547251685.194911:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251685.194913:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x800ad [1] + 192 00000400:00000200:13.0:1547251685.194915:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05170 00000100:00000200:15.0:1547251685.194916:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b6300 x1622384012448128/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251692 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251685.194928:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251685.194930:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (22)++ 00000400:00000200:13.0:1547251685.194930:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x800bd [1] + 192 00000800:00000200:15.0:1547251685.194931:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:15.0:1547251685.194932:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:15.0:1547251685.194933:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:13.0:1547251685.194933:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960eaa00 x1622384012448112/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251692 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251685.194935:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:15.0:1547251685.194936:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:15.0:1547251685.194938:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000400:00000200:15.0:1547251685.194941:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251685.194952:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05190 00000400:00000200:13.0:1547251685.194952:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251685.194954:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:1.0:1547251685.194954:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b6300 x1622384012448128/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251692 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251685.194955:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:13.0:1547251685.194956:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:13.0:1547251685.194958:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000100:00000200:1.0:1547251685.194962:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b6300 x1622384012448128/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251692 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251685.194974:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00000100:00000200:10.0:1547251685.194977:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960eaa00 x1622384012448112/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251692 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:1.0:1547251685.194979:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b6300 x1622384012448128/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251692 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:10.0:1547251685.194983:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960eaa00 x1622384012448112/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251692 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251685.194987:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914aa0 00000400:00000200:15.0:1547251685.194989:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x800cd [1] + 192 00010000:00080000:10.0:1547251685.194991:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba960eaa00 x1622384012448112/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251692 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:15.0:1547251685.194992:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df4500 x1622384012448144/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251692 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251685.195000:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251685.195001:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (21)++ 00000800:00000200:15.0:1547251685.195002:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (22)-- 00000800:00000200:15.0:1547251685.195003:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:4.0:1547251685.195010:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df4500 x1622384012448144/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251692 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251685.195015:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df4500 x1622384012448144/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251692 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251685.195019:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b110 00010000:00080000:4.0:1547251685.195023:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a765df4500 x1622384012448144/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251692 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:12.0:1547251687.306680:0:12684:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448160, portal 4 00000100:00000200:12.0:1547251687.306683:0:12684:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448160, offset 0 00000400:00000200:12.0:1547251687.306686:0:12684:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:12.0:1547251687.306689:0:12684:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:12.0:1547251687.306692:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23101] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251687.306695:0:12684:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23101] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23101] 00000400:00000200:12.0:1547251687.306697:0:12684:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:12.0:1547251687.306699:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251687.306700:0:12684:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251687.306701:0:12684:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9778, 9778] 00000400:00000200:12.0:1547251687.306703:0:12684:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:12.0:1547251687.306708:0:12684:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251687.306710:0:12684:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000100:00000200:9.0:1547251687.306712:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448176, portal 4 00000800:00000200:12.0:1547251687.306713:0:12684:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:12.0:1547251687.306715:0:12684:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000100:00000200:9.0:1547251687.306715:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448176, offset 0 00000800:00000200:12.0:1547251687.306716:0:12684:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000400:00000200:9.0:1547251687.306718:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000800:00000200:12.0:1547251687.306719:0:12684:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000400:00000200:9.0:1547251687.306721:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251687.306723:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23102] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251687.306727:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23101] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23102] 00000400:00000200:9.0:1547251687.306729:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251687.306730:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251687.306732:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251687.306733:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9778, 9779] 00000400:00000200:9.0:1547251687.306735:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:9.0:1547251687.306739:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251687.306742:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:9.0:1547251687.306745:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251687.306746:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:9.0:1547251687.306747:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:9.0:1547251687.306750:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000100:00000200:0.0:1547251687.306755:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448192, portal 4 00000100:00000200:0.0:1547251687.306759:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448192, offset 0 00000400:00000200:0.0:1547251687.306763:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:0.0:1547251687.306766:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:0.0:1547251687.306768:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23102] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:5.2:1547251687.306769:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000400:00000200:0.0:1547251687.306771:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23102] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23102] 00000400:00000200:0.0:1547251687.306773:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:0.0:1547251687.306775:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251687.306776:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251687.306777:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9779, 9779] 00000400:00000200:0.0:1547251687.306779:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251687.306784:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000400:00000200:0.0:1547251687.306784:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:0.0:1547251687.306786:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:0.0:1547251687.306789:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251687.306791:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (23)++ 00000800:00000200:0.0:1547251687.306792:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (24)++ 00000800:00000200:0.0:1547251687.306795:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (25)-- 00000800:00000200:10.0:1547251687.306796:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (24)-- 00000800:00000200:7.2:1547251687.306799:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000100:00000200:10.0:1547251687.306800:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f2a00 x1622384012448160/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251687.306808:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038110 00000400:00000200:10.0:1547251687.306810:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251687.306811:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (23)++ 00000800:00000200:11.0:1547251687.306814:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (24)-- 00000400:00000200:10.0:1547251687.306814:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251687.306817:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (23)-- 00000100:00000200:11.0:1547251687.306818:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c1800 x1622384012448192/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251687.306819:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251687.306820:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000400:00000200:11.0:1547251687.306824:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246656e8 00000100:00000200:10.0:1547251687.306824:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf6600 x1622384012448176/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251687.306825:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251687.306828:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251687.306829:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaee0 00000800:00000200:11.0:1547251687.306831:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251687.306831:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251687.306832:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:11.0:1547251687.306834:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:10.0:1547251687.306834:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251687.306836:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:5.2:1547251687.306905:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251687.306910:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:5.2:1547251687.306917:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:11.0:1547251687.306919:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251687.306923:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000800:00000200:10.0:1547251687.306924:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000800:00000200:10.0:1547251687.306926:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:11.0:1547251687.306928:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251687.306931:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251687.306933:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c051b0 00000400:00000200:10.0:1547251687.306934:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c051a0 00000400:00000200:11.0:1547251687.306937:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd77b9 [1] + 192 00000400:00000200:10.0:1547251687.306942:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd77a9 [1] + 192 00000100:00000200:11.0:1547251687.306954:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf6600 x1622384012448176/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251687.306966:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251687.306968:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000100:00000200:10.0:1547251687.306969:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f2a00 x1622384012448160/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251687.306970:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251687.306971:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251687.306973:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251687.306976:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251687.306979:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:11.0:1547251687.306980:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251687.306981:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000100:00000200:9.0:1547251687.306981:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf6600 x1622384012448176/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251687.306983:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)-- 00000400:00000200:11.0:1547251687.306984:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c051c0 00000800:00000200:10.0:1547251687.306984:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:10.0:1547251687.306986:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:9.0:1547251687.306986:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf6600 x1622384012448176/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251687.306987:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000800:00000200:10.0:1547251687.306989:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:9.0:1547251687.306993:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eac38 00000400:00000200:11.0:1547251687.306996:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd77d9 [1] + 192 00010000:00080000:9.0:1547251687.306996:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf6600 x1622384012448176/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:12.0:1547251687.307001:0:12684:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f2a00 x1622384012448160/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:12.0:1547251687.307007:0:12684:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f2a00 x1622384012448160/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251687.307011:0:12684:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038990 00000100:00000200:11.0:1547251687.307013:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c1800 x1622384012448192/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:12.0:1547251687.307015:0:12684:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f2a00 x1622384012448160/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:11.0:1547251687.307023:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251687.307024:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251687.307025:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251687.307026:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000100:00000200:0.0:1547251687.307033:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c1800 x1622384012448192/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251687.307039:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c1800 x1622384012448192/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251687.307043:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00010000:00080000:0.0:1547251687.307046:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c1800 x1622384012448192/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251687.770694:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448208, portal 4 00000100:00000200:12.0:1547251687.770696:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448224, portal 4 00000100:00000200:9.0:1547251687.770697:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448208, offset 0 00000100:00000200:12.0:1547251687.770699:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448224, offset 0 00000400:00000200:9.0:1547251687.770700:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:12.0:1547251687.770703:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251687.770703:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251687.770705:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23103] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:1.0:1547251687.770705:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448240, portal 4 00000400:00000200:12.0:1547251687.770706:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000100:00000200:1.0:1547251687.770708:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448240, offset 0 00000400:00000200:12.0:1547251687.770709:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23103] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251687.770709:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23102] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23103] 00000400:00000200:9.0:1547251687.770711:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:1.0:1547251687.770711:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:12.0:1547251687.770712:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23102] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23103] 00000400:00000200:9.0:1547251687.770713:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:12.0:1547251687.770714:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251687.770714:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251687.770714:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:12.0:1547251687.770716:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251687.770716:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9771, 9772] 00000400:00000200:1.0:1547251687.770716:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23104] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:12.0:1547251687.770718:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251687.770718:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:12.0:1547251687.770719:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9771, 9772] 00000400:00000200:1.0:1547251687.770719:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23103] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23104] 00000400:00000200:12.0:1547251687.770721:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:1.0:1547251687.770722:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251687.770723:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000400:00000200:1.0:1547251687.770723:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:9.0:1547251687.770725:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:1.0:1547251687.770725:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:1.0:1547251687.770726:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9772, 9773] 00000400:00000200:1.0:1547251687.770728:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000800:00000200:9.0:1547251687.770729:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:12.0:1547251687.770730:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:9.0:1547251687.770730:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:9.0:1547251687.770731:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:12.0:1547251687.770733:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:9.0:1547251687.770734:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:12.0:1547251687.770736:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:1.0:1547251687.770737:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000800:00000200:12.0:1547251687.770738:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000800:00000200:12.0:1547251687.770739:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000800:00000200:1.0:1547251687.770741:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:12.0:1547251687.770742:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000800:00000200:1.0:1547251687.770745:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:1.0:1547251687.770748:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:12.2:1547251687.770750:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:1.0:1547251687.770750:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:1.0:1547251687.770753:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:0.2:1547251687.770757:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:0.2:1547251687.770765:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:4.0:1547251687.770808:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251687.770812:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:4.0:1547251687.770812:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251687.770814:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000100:00000200:4.0:1547251687.770816:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6d04f2d00 x1622384012448224/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251687.770817:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf0600 x1622384012448208/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251687.770823:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038198 00000400:00000200:5.0:1547251687.770824:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea770 00000400:00000200:5.0:1547251687.770825:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251687.770825:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251687.770829:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251687.770829:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251687.770832:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251687.770833:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251687.770834:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:5.0:1547251687.770835:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:4.0:1547251687.770836:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:5.0:1547251687.770837:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:4.0:1547251687.770838:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b6000 x1622384012448240/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251687.770839:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:4.0:1547251687.770844:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2198 00000400:00000200:4.0:1547251687.770845:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251687.770848:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251687.770850:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:0.2:1547251687.770912:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:12.2:1547251687.770918:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:5.0:1547251687.770926:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251687.770930:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251687.770932:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000400:00000200:5.0:1547251687.770935:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251687.770935:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251687.770939:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251687.770940:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c051d0 00000400:00000200:4.0:1547251687.770952:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c051e0 00000400:00000200:5.0:1547251687.770954:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4735 [1] + 192 00000400:00000200:4.0:1547251687.770958:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4745 [1] + 192 00000100:00000200:5.0:1547251687.770962:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf0600 x1622384012448208/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251687.770975:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251687.770976:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6d04f2d00 x1622384012448224/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251687.770977:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251687.770979:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251687.770981:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251687.770982:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:5.0:1547251687.770984:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251687.770986:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:4.0:1547251687.770987:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251687.770989:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf0600 x1622384012448208/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251687.770989:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (23)++ 00000400:00000200:5.0:1547251687.770990:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251687.770990:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (24)-- 00000800:00000200:4.0:1547251687.770991:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251687.770993:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000100:00000200:9.0:1547251687.770995:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf0600 x1622384012448208/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251687.770995:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c051f0 00000400:00000200:9.0:1547251687.770999:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eac38 00000100:00000200:12.0:1547251687.771002:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6d04f2d00 x1622384012448224/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:9.0:1547251687.771002:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf0600 x1622384012448208/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:12.0:1547251687.771008:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6d04f2d00 x1622384012448224/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:12.0:1547251687.771012:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b17b038990 00000400:00000200:5.0:1547251687.771015:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4755 [1] + 192 00010000:00080000:12.0:1547251687.771016:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6d04f2d00 x1622384012448224/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251687.771018:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b6000 x1622384012448240/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251687.771026:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251687.771027:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251687.771028:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251687.771029:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:1.0:1547251687.771036:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b6000 x1622384012448240/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:1.0:1547251687.771042:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b6000 x1622384012448240/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251687.771046:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00010000:00080000:1.0:1547251687.771050:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b6000 x1622384012448240/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251694 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251688.170687:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448256, portal 4 00000100:00000200:9.0:1547251688.170689:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448256, offset 0 00000400:00000200:9.0:1547251688.170693:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251688.170696:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251688.170698:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c05200 00000400:00000200:9.0:1547251688.170701:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4e89 [1] + 0 00000400:00000200:9.0:1547251688.170703:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ae6a914e58 00000100:00000200:9.0:1547251688.170705:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251688.170713:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914e58 00000400:00000200:9.0:1547251688.170714:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:9.0:1547251688.170718:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf0300 x1622384012448256/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251695 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251688.170725:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea990 00000400:00000200:9.0:1547251688.170726:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251688.170727:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251688.170736:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012448256 00010000:00000200:8.0:1547251688.170745:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c2df850 x1622384012448256/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:749/0 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:8.0:1547251688.170752:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012448256, offset 192 00000400:00000200:8.0:1547251688.170754:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251688.170757:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:5.0:1547251688.170757:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448272, portal 4 00000400:00000200:8.0:1547251688.170759:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c05200 00000100:00000200:5.0:1547251688.170760:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448272, offset 0 00000400:00000200:8.0:1547251688.170762:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf6c1 [1] + 192 00000400:00000200:5.0:1547251688.170763:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:8.0:1547251688.170764:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf0300 x1622384012448256/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251695 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251688.170766:0:12674:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251688.170768:0:12674:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c05210 00000400:00000200:5.0:1547251688.170772:0:12674:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2f35 [1] + 0 00000400:00000200:8.0:1547251688.170773:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251688.170774:0:12674:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad220 00000400:00000200:8.0:1547251688.170775:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688aa0 00000400:00000200:8.0:1547251688.170776:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:5.0:1547251688.170776:0:12674:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251688.170779:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:9.0:1547251688.170783:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf0300 x1622384012448256/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251695 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251688.170783:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad220 00000400:00000200:5.0:1547251688.170784:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:8.0:1547251688.170785:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251688.170787:0:12674:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae37932100 x1622384012448272/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251695 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251688.170789:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf0300 x1622384012448256/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251695 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251688.170791:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad770 00000400:00000200:5.0:1547251688.170792:0:12674:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251688.170793:0:12674:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:9.0:1547251688.170794:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eac38 00010000:00080000:9.0:1547251688.170798:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf0300 x1622384012448256/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251695 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:6.0:1547251688.170807:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012448272 00010000:00000200:6.0:1547251688.170818:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11cc50 x1622384012448272/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:749/0 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251688.170827:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012448272, offset 192 00000400:00000200:6.0:1547251688.170831:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251688.170834:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251688.170837:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c05210 00000400:00000200:6.0:1547251688.170841:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf6d1 [1] + 192 00000100:00000200:6.0:1547251688.170844:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae37932100 x1622384012448272/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251695 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251688.170853:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251688.170855:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990d48 00000400:00000200:6.0:1547251688.170856:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251688.170858:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251688.170861:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae37932100 x1622384012448272/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251695 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251688.170863:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251688.170866:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae37932100 x1622384012448272/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251695 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251688.170870:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad330 00010000:00080000:5.0:1547251688.170873:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae37932100 x1622384012448272/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251695 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251688.186719:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448288, portal 4 00000100:00000200:4.0:1547251688.186724:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448288, offset 0 00000400:00000200:4.0:1547251688.186727:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251688.186730:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251688.186733:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c05220 00000400:00000200:4.0:1547251688.186735:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe2f65 [1] + 0 00000400:00000200:4.0:1547251688.186737:0:12675:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073add48 00000100:00000200:4.0:1547251688.186739:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:4.0:1547251688.186748:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073add48 00000400:00000200:4.0:1547251688.186749:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251688.186751:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df7800 x1622384012448288/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251695 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251688.186756:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b000 00000400:00000200:4.0:1547251688.186758:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251688.186759:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:6.0:1547251688.186769:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012448288 00010000:00000200:6.0:1547251688.186781:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11c850 x1622384012448288/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:749/0 lens 224/368 e 0 to 0 dl 1547251694 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:6.0:1547251688.186790:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012448288, offset 192 00000400:00000200:6.0:1547251688.186793:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:6.0:1547251688.186797:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:6.0:1547251688.186800:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c05220 00000400:00000200:6.0:1547251688.186803:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf6e1 [1] + 192 00000100:00000200:6.0:1547251688.186806:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df7800 x1622384012448288/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251695 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251688.186815:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:6.0:1547251688.186817:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b6459907f8 00000400:00000200:6.0:1547251688.186818:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:6.0:1547251688.186819:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251688.186824:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df7800 x1622384012448288/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251695 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251688.186825:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:4.0:1547251688.186829:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df7800 x1622384012448288/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251695 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251688.186832:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b110 00010000:00080000:4.0:1547251688.186835:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df7800 x1622384012448288/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251695 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:0.0:1547251690.202681:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448304, portal 4 00000100:00000200:0.0:1547251690.202684:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448304, offset 0 00000400:00000200:0.0:1547251690.202688:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:0.0:1547251690.202690:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:0.0:1547251690.202693:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23104] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251690.202696:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23104] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23104] 00000100:00000200:11.0:1547251690.202697:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448320, portal 4 00000400:00000200:0.0:1547251690.202698:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:0.0:1547251690.202699:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:11.0:1547251690.202701:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448320, offset 0 00000400:00000200:0.0:1547251690.202701:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:0.0:1547251690.202702:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9773, 9774] 00000400:00000200:11.0:1547251690.202704:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:0.0:1547251690.202704:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:11.0:1547251690.202706:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:0.0:1547251690.202708:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:11.0:1547251690.202709:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23105] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:0.0:1547251690.202711:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:11.0:1547251690.202712:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23104] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23105] 00000400:00000200:11.0:1547251690.202714:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:0.0:1547251690.202714:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251690.202715:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000400:00000200:11.0:1547251690.202716:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:0.0:1547251690.202716:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:11.0:1547251690.202717:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:11.0:1547251690.202719:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9774, 9774] 00000800:00000200:0.0:1547251690.202719:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:11.0:1547251690.202720:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:11.0:1547251690.202725:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:11.0:1547251690.202728:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:11.0:1547251690.202731:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:11.0:1547251690.202732:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:11.0:1547251690.202733:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:11.0:1547251690.202736:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:9.2:1547251690.202751:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:7.2:1547251690.202760:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:4.0:1547251690.202799:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448336, portal 4 00000100:00000200:4.0:1547251690.202803:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448336, offset 0 00000800:00000200:13.0:1547251690.202806:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000400:00000200:4.0:1547251690.202807:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:13.0:1547251690.202809:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000400:00000200:4.0:1547251690.202809:0:12677:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000100:00000200:13.0:1547251690.202812:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b17b07ec00 x1622384012448320/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251697 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251690.202812:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23105] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:4.0:1547251690.202815:0:12677:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23105] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23105] 00000400:00000200:4.0:1547251690.202817:0:12677:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:13.0:1547251690.202818:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0bbb0 00000400:00000200:13.0:1547251690.202819:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251690.202819:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:4.0:1547251690.202820:0:12677:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:15.0:1547251690.202821:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000400:00000200:4.0:1547251690.202821:0:12677:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9774, 9775] 00000400:00000200:13.0:1547251690.202822:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:4.0:1547251690.202823:0:12677:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000800:00000200:15.0:1547251690.202824:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:13.0:1547251690.202824:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:13.0:1547251690.202826:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000100:00000200:15.0:1547251690.202827:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328df200 x1622384012448304/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251697 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251690.202827:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:4.0:1547251690.202828:0:12677:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:4.0:1547251690.202831:0:12677:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:15.0:1547251690.202833:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665dd0 00000400:00000200:15.0:1547251690.202834:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251690.202834:0:12677:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000400:00000200:15.0:1547251690.202836:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:4.0:1547251690.202836:0:12677:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:4.0:1547251690.202837:0:12677:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:15.0:1547251690.202839:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:4.0:1547251690.202839:0:12677:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:7.2:1547251690.202853:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:13.0:1547251690.202866:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:13.0:1547251690.202874:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:13.0:1547251690.202876:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df0600 x1622384012448336/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251697 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251690.202881:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:13.0:1547251690.202881:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b000 00000400:00000200:13.0:1547251690.202882:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251690.202884:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251690.202886:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:7.2:1547251690.202890:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251690.202895:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:15.0:1547251690.202901:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:13.0:1547251690.202902:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:15.0:1547251690.202904:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000800:00000200:13.0:1547251690.202904:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251690.202907:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251690.202908:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251690.202913:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05240 00000400:00000200:13.0:1547251690.202913:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05230 00000400:00000200:15.0:1547251690.202916:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x8010d [1] + 192 00000400:00000200:13.0:1547251690.202920:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x800fd [1] + 192 00000100:00000200:15.0:1547251690.202923:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b17b07ec00 x1622384012448320/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251697 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251690.202935:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:13.0:1547251690.202936:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328df200 x1622384012448304/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251697 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251690.202937:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000800:00000200:15.0:1547251690.202938:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:15.0:1547251690.202939:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:15.0:1547251690.202941:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:15.0:1547251690.202942:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:15.0:1547251690.202951:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:13.0:1547251690.202952:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:15.0:1547251690.202954:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000800:00000200:13.0:1547251690.202954:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)++ 00000800:00000200:13.0:1547251690.202956:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (24)-- 00000100:00000200:11.0:1547251690.202956:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b17b07ec00 x1622384012448320/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251697 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251690.202957:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05250 00000800:00000200:13.0:1547251690.202957:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:13.0:1547251690.202958:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:11.0:1547251690.202961:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b17b07ec00 x1622384012448320/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251697 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251690.202966:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a475d0b440 00000400:00000200:15.0:1547251690.202968:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x8011d [1] + 192 00010000:00080000:11.0:1547251690.202969:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b17b07ec00 x1622384012448320/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251697 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:0.0:1547251690.202972:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328df200 x1622384012448304/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251697 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:0.0:1547251690.202977:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328df200 x1622384012448304/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251697 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251690.202981:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00000100:00000200:15.0:1547251690.202983:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df0600 x1622384012448336/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251697 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:0.0:1547251690.202985:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328df200 x1622384012448304/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251697 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:15.0:1547251690.202991:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251690.202993:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:15.0:1547251690.202994:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:15.0:1547251690.202995:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000100:00000200:4.0:1547251690.203001:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df0600 x1622384012448336/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251697 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251690.203007:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df0600 x1622384012448336/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251697 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251690.203011:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b110 00010000:00080000:4.0:1547251690.203015:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a765df0600 x1622384012448336/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251697 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:1.0:1547251692.314675:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448352, portal 4 00000100:00000200:1.0:1547251692.314678:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448352, offset 0 00000400:00000200:1.0:1547251692.314682:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:1.0:1547251692.314685:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:1.0:1547251692.314687:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23106] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:1.0:1547251692.314690:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23105] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23106] 00000400:00000200:1.0:1547251692.314693:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:1.0:1547251692.314694:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:8.0:1547251692.314695:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448384, portal 4 00000400:00000200:1.0:1547251692.314695:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:8.0:1547251692.314697:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448384, offset 0 00000400:00000200:1.0:1547251692.314697:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9779, 9780] 00000400:00000200:1.0:1547251692.314698:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:8.0:1547251692.314700:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:1.0:1547251692.314703:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:1.0:1547251692.314706:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:8.0:1547251692.314707:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000800:00000200:1.0:1547251692.314709:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000400:00000200:8.0:1547251692.314710:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23106] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:1.0:1547251692.314710:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:1.0:1547251692.314711:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000400:00000200:8.0:1547251692.314713:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23106] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23106] 00000800:00000200:1.0:1547251692.314714:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000400:00000200:8.0:1547251692.314715:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251692.314716:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251692.314718:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251692.314719:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 8], s:[9780, 9780] 00000400:00000200:8.0:1547251692.314721:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:8.0:1547251692.314725:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251692.314728:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:8.0:1547251692.314731:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251692.314733:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000100:00000200:14.0:1547251692.314734:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448368, portal 4 00000800:00000200:8.0:1547251692.314735:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:8.0:1547251692.314737:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000100:00000200:14.0:1547251692.314739:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448368, offset 0 00000400:00000200:14.0:1547251692.314743:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:14.0:1547251692.314748:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:14.0:1547251692.314750:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23107] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:14.0:1547251692.314753:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23106] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23107] 00000400:00000200:14.0:1547251692.314755:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:14.0:1547251692.314757:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251692.314758:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251692.314759:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[7, 7], s:[9780, 9781] 00000400:00000200:14.0:1547251692.314761:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000800:00000200:7.2:1547251692.314763:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000400:00000200:14.0:1547251692.314766:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251692.314769:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:14.0:1547251692.314772:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251692.314773:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (22)++ 00000800:00000200:14.0:1547251692.314774:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (23)++ 00000800:00000200:14.0:1547251692.314777:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (24)-- 00000800:00000200:5.2:1547251692.314787:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000800:00000200:10.0:1547251692.314797:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:10.0:1547251692.314801:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000800:00000200:11.0:1547251692.314804:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000100:00000200:10.0:1547251692.314806:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b5d00 x1622384012448352/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251692.314807:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000100:00000200:11.0:1547251692.314810:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824756300 x1622384012448384/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251692.314818:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688880 00000400:00000200:11.0:1547251692.314819:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251692.314819:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e25d8 00000400:00000200:10.0:1547251692.314820:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251692.314823:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251692.314824:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251692.314826:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251692.314832:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251692.314833:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251692.314834:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:11.0:1547251692.314835:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251692.314836:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000100:00000200:11.0:1547251692.314837:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af0600 x1622384012448368/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251692.314843:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:11.0:1547251692.314844:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251692.314847:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251692.314848:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:7.2:1547251692.314854:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:10.0:1547251692.314867:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251692.314876:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.1.11.91@o2ib10 00000800:00000200:10.0:1547251692.314879:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251692.314880:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251692.314881:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:10.0:1547251692.314882:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251692.314883:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000800:00000200:10.0:1547251692.314883:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000800:00000200:5.2:1547251692.314886:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000400:00000200:11.0:1547251692.314888:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251692.314893:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05260 00000800:00000200:10.0:1547251692.314894:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000400:00000200:11.0:1547251692.314897:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7839 [1] + 192 00000800:00000200:10.0:1547251692.314897:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000800:00000200:7.2:1547251692.314899:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000400:00000200:10.0:1547251692.314901:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000100:00000200:11.0:1547251692.314902:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b5d00 x1622384012448352/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251692.314905:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05280 00000400:00000200:11.0:1547251692.314913:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251692.314915:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000400:00000200:10.0:1547251692.314915:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7849 [1] + 192 00000800:00000200:11.0:1547251692.314917:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251692.314918:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000100:00000200:10.0:1547251692.314919:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824756300 x1622384012448384/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251692.314920:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251692.314921:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251692.314924:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251692.314927:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251692.314930:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:11.0:1547251692.314931:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05270 00000800:00000200:10.0:1547251692.314931:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (21)++ 00000100:00000200:1.0:1547251692.314932:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b5d00 x1622384012448352/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251692.314933:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251692.314934:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:10.0:1547251692.314936:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000100:00000200:1.0:1547251692.314937:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b5d00 x1622384012448352/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:1.0:1547251692.314941:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8246e2440 00000400:00000200:11.0:1547251692.314953:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7869 [1] + 192 00010000:00080000:1.0:1547251692.314954:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b5d00 x1622384012448352/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251692.314957:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824756300 x1622384012448384/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251692.314963:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824756300 x1622384012448384/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251692.314967:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688dd0 00000100:00000200:11.0:1547251692.314969:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af0600 x1622384012448368/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251692.314970:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824756300 x1622384012448384/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:11.0:1547251692.314977:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251692.314978:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (21)++ 00000800:00000200:11.0:1547251692.314979:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (22)-- 00000800:00000200:11.0:1547251692.314980:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000100:00000200:14.0:1547251692.314987:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af0600 x1622384012448368/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:14.0:1547251692.314992:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af0600 x1622384012448368/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251692.314996:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00010000:00080000:14.0:1547251692.315000:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af0600 x1622384012448368/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251692.778679:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448400, portal 4 00000100:00000200:8.0:1547251692.778682:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448400, offset 0 00000400:00000200:8.0:1547251692.778685:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251692.778688:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251692.778690:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23107] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251692.778693:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23107] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23107] 00000400:00000200:8.0:1547251692.778695:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251692.778697:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251692.778698:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251692.778699:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9772, 9774] 00000400:00000200:8.0:1547251692.778701:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:8.0:1547251692.778706:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251692.778709:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000100:00000200:0.0:1547251692.778710:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448432, portal 4 00000800:00000200:8.0:1547251692.778712:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251692.778713:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000100:00000200:0.0:1547251692.778713:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448432, offset 0 00000800:00000200:8.0:1547251692.778714:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000400:00000200:0.0:1547251692.778716:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:8.0:1547251692.778717:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000400:00000200:0.0:1547251692.778719:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:0.0:1547251692.778721:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23108] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:0.0:1547251692.778724:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23107] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23108] 00000400:00000200:0.0:1547251692.778726:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:0.0:1547251692.778728:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:15.0:1547251692.778730:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448416, portal 4 00000400:00000200:0.0:1547251692.778730:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:12.2:1547251692.778731:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000400:00000200:0.0:1547251692.778731:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9773, 9774] 00000400:00000200:0.0:1547251692.778733:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000100:00000200:15.0:1547251692.778734:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448416, offset 0 00000400:00000200:15.0:1547251692.778738:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:0.0:1547251692.778738:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:15.0:1547251692.778741:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:0.0:1547251692.778741:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:15.0:1547251692.778743:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23108] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:0.0:1547251692.778744:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000800:00000200:0.0:1547251692.778745:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000400:00000200:15.0:1547251692.778746:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23108] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23108] 00000800:00000200:0.0:1547251692.778746:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000400:00000200:15.0:1547251692.778748:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000800:00000200:0.0:1547251692.778749:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000400:00000200:15.0:1547251692.778750:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251692.778751:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251692.778752:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9773, 9775] 00000400:00000200:15.0:1547251692.778754:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:15.0:1547251692.778759:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:0.2:1547251692.778760:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:15.0:1547251692.778762:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:15.0:1547251692.778765:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251692.778767:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:15.0:1547251692.778768:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (23)++ 00000800:00000200:15.0:1547251692.778771:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (24)-- 00000800:00000200:5.0:1547251692.778790:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (23)++ 00000800:00000200:5.0:1547251692.778793:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (24)-- 00000800:00000200:4.0:1547251692.778793:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251692.778796:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000100:00000200:5.0:1547251692.778798:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824755400 x1622384012448400/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251692.778799:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328df500 x1622384012448432/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251692.778805:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688880 00000400:00000200:4.0:1547251692.778805:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665000 00000400:00000200:5.0:1547251692.778807:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251692.778807:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251692.778811:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251692.778811:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251692.778813:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251692.778815:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251692.778816:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:4.0:1547251692.778816:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:4.0:1547251692.778817:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251692.778818:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000100:00000200:4.0:1547251692.778820:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97bae1af7800 x1622384012448416/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251692.778826:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1bb0 00000400:00000200:4.0:1547251692.778827:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251692.778830:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251692.778831:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:12.2:1547251692.778893:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:0.2:1547251692.778898:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:5.0:1547251692.778908:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251692.778911:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000800:00000200:4.0:1547251692.778911:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:4.0:1547251692.778914:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251692.778916:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251692.778918:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251692.778921:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c052b0 00000400:00000200:4.0:1547251692.778922:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05290 00000400:00000200:5.0:1547251692.778925:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe47e5 [1] + 192 00000400:00000200:4.0:1547251692.778932:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe47d5 [1] + 192 00000100:00000200:5.0:1547251692.778936:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328df500 x1622384012448432/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251692.778958:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251692.778960:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (22)++ 00000100:00000200:4.0:1547251692.778960:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824755400 x1622384012448400/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251692.778962:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251692.778964:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:5.0:1547251692.778965:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:5.0:1547251692.778967:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251692.778970:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251692.778971:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:4.0:1547251692.778972:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)++ 00000100:00000200:0.0:1547251692.778973:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328df500 x1622384012448432/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251692.778974:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251692.778974:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (24)-- 00000800:00000200:4.0:1547251692.778975:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:5.0:1547251692.778977:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c052a0 00000800:00000200:4.0:1547251692.778977:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:0.0:1547251692.778978:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328df500 x1622384012448432/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:0.0:1547251692.778983:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824665e58 00000400:00000200:5.0:1547251692.778986:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe47f5 [1] + 192 00010000:00080000:0.0:1547251692.778986:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328df500 x1622384012448432/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251692.778990:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824755400 x1622384012448400/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251692.778996:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824755400 x1622384012448400/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251692.779005:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688dd0 00000100:00000200:5.0:1547251692.779007:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97bae1af7800 x1622384012448416/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251692.779009:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824755400 x1622384012448400/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251692.779016:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251692.779017:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251692.779018:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:5.0:1547251692.779019:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:15.0:1547251692.779026:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97bae1af7800 x1622384012448416/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:15.0:1547251692.779032:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97bae1af7800 x1622384012448416/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251692.779036:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb15d8 00010000:00080000:15.0:1547251692.779040:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97bae1af7800 x1622384012448416/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251699 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251693.178672:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448448, portal 4 00000100:00000200:8.0:1547251693.178675:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448448, offset 0 00000400:00000200:8.0:1547251693.178677:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251693.178680:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251693.178683:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c052c0 00000400:00000200:8.0:1547251693.178685:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4ef9 [1] + 0 00000400:00000200:8.0:1547251693.178687:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ba9d367bb0 00000100:00000200:8.0:1547251693.178689:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:8.0:1547251693.178698:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba9d367bb0 00000400:00000200:8.0:1547251693.178699:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251693.178700:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448464, portal 4 00000100:00000200:8.0:1547251693.178703:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824756f00 x1622384012448448/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251700 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251693.178703:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448464, offset 0 00000400:00000200:4.0:1547251693.178706:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251693.178709:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688660 00000400:00000200:4.0:1547251693.178709:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251693.178710:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:8.0:1547251693.178712:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:4.0:1547251693.178712:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c052d0 00000400:00000200:4.0:1547251693.178714:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe3085 [1] + 0 00000400:00000200:4.0:1547251693.178716:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073adb28 00000100:00000200:4.0:1547251693.178718:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:9.0:1547251693.178720:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012448448 00000400:00000200:4.0:1547251693.178727:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073adb28 00000400:00000200:4.0:1547251693.178728:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00010000:00000200:9.0:1547251693.178730:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c2df450 x1622384012448448/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:754/0 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:4.0:1547251693.178731:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c431b00 x1622384012448464/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251700 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251693.178736:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040baa0 00000400:00000200:4.0:1547251693.178736:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:9.0:1547251693.178737:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012448448, offset 192 00000400:00000200:4.0:1547251693.178737:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:9.0:1547251693.178739:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251693.178741:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251693.178744:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c052c0 00000400:00000200:9.0:1547251693.178746:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf721 [1] + 192 00000100:00000200:9.0:1547251693.178748:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824756f00 x1622384012448448/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251700 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251693.178757:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251693.178759:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eac38 00000100:00000200:7.0:1547251693.178759:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012448464 00000400:00000200:9.0:1547251693.178760:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:9.0:1547251693.178764:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:8.0:1547251693.178766:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824756f00 x1622384012448448/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251700 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00000200:7.0:1547251693.178768:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11c450 x1622384012448464/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:754/0 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:9.0:1547251693.178771:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:8.0:1547251693.178772:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824756f00 x1622384012448448/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251700 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251693.178775:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012448464, offset 192 00000400:00000200:7.0:1547251693.178777:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251693.178778:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688dd0 00010000:00080000:8.0:1547251693.178782:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b824756f00 x1622384012448448/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251700 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:7.0:1547251693.178782:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251693.178785:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c052d0 00000400:00000200:7.0:1547251693.178787:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf731 [1] + 192 00000100:00000200:7.0:1547251693.178790:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c431b00 x1622384012448464/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251700 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251693.178800:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251693.178802:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17ea880 00000400:00000200:7.0:1547251693.178803:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251693.178804:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251693.178807:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c431b00 x1622384012448464/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251700 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251693.178809:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:4.0:1547251693.178812:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c431b00 x1622384012448464/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251700 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251693.178815:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b110 00010000:00080000:4.0:1547251693.178818:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ae5c431b00 x1622384012448464/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251700 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:4.0:1547251693.194729:0:12675:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448480, portal 4 00000100:00000200:4.0:1547251693.194732:0:12675:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448480, offset 0 00000400:00000200:4.0:1547251693.194736:0:12675:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:4.0:1547251693.194739:0:12675:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:4.0:1547251693.194742:0:12675:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c052e0 00000400:00000200:4.0:1547251693.194744:0:12675:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe30b5 [1] + 0 00000400:00000200:4.0:1547251693.194746:0:12675:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040ba18 00000100:00000200:4.0:1547251693.194748:0:12675:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:4.0:1547251693.194755:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040ba18 00000400:00000200:4.0:1547251693.194755:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:4.0:1547251693.194758:0:12675:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a765df6600 x1622384012448480/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251700 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251693.194762:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040baa0 00000400:00000200:4.0:1547251693.194764:0:12675:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251693.194765:0:12675:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251693.194783:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012448480 00010000:00000200:7.0:1547251693.194792:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11c050 x1622384012448480/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:754/0 lens 224/368 e 0 to 0 dl 1547251699 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:7.0:1547251693.194798:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012448480, offset 192 00000400:00000200:7.0:1547251693.194801:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251693.194804:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251693.194807:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c052e0 00000400:00000200:7.0:1547251693.194809:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf741 [1] + 192 00000100:00000200:7.0:1547251693.194812:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a765df6600 x1622384012448480/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251700 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251693.194822:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251693.194824:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eaa18 00000400:00000200:7.0:1547251693.194825:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251693.194826:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:4.0:1547251693.194829:0:12675:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a765df6600 x1622384012448480/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251700 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251693.194831:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:4.0:1547251693.194834:0:12675:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a765df6600 x1622384012448480/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251700 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251693.194837:0:12675:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b110 00010000:00080000:4.0:1547251693.194840:0:12675:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97a765df6600 x1622384012448480/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251700 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251695.210708:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448512, portal 4 00000100:00000200:2.0:1547251695.210710:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448496, portal 4 00000100:00000200:5.0:1547251695.210711:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448512, offset 0 00000100:00000200:2.0:1547251695.210713:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448496, offset 0 00000400:00000200:5.0:1547251695.210714:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:5.0:1547251695.210717:0:12674:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:2.0:1547251695.210717:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:5.0:1547251695.210719:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23109] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251695.210720:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:5.0:1547251695.210723:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23108] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23109] 00000400:00000200:2.0:1547251695.210723:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23109] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:5.0:1547251695.210725:0:12674:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251695.210726:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23108] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23109] 00000400:00000200:5.0:1547251695.210727:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:5.0:1547251695.210729:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251695.210729:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:5.0:1547251695.210730:0:12674:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9775, 9775] 00000400:00000200:2.0:1547251695.210730:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:5.0:1547251695.210732:0:12674:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:2.0:1547251695.210732:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251695.210733:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9775, 9775] 00000400:00000200:2.0:1547251695.210735:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000100:00000200:10.0:1547251695.210737:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448528, portal 4 00000400:00000200:5.0:1547251695.210737:0:12674:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000100:00000200:10.0:1547251695.210739:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448528, offset 0 00000800:00000200:5.0:1547251695.210739:0:12674:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:10.0:1547251695.210743:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:5.0:1547251695.210744:0:12674:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000400:00000200:2.0:1547251695.210744:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000400:00000200:10.0:1547251695.210745:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000800:00000200:5.0:1547251695.210745:0:12674:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (20)++ 00000800:00000200:5.0:1547251695.210746:0:12674:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:2.0:1547251695.210747:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:10.0:1547251695.210748:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23110] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:5.0:1547251695.210749:0:12674:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:2.0:1547251695.210750:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000400:00000200:10.0:1547251695.210751:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:255, d:10, s:23109] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23110] 00000800:00000200:2.0:1547251695.210752:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000400:00000200:10.0:1547251695.210753:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:2.0:1547251695.210753:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:10.0:1547251695.210756:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:2.0:1547251695.210756:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:10.0:1547251695.210757:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251695.210759:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9776, 9776] 00000400:00000200:10.0:1547251695.210760:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000400:00000200:10.0:1547251695.210765:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000800:00000200:10.0:1547251695.210768:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000800:00000200:10.0:1547251695.210771:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4380] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:9.2:1547251695.210771:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:10.0:1547251695.210772:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (21)++ 00000800:00000200:10.0:1547251695.210773:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:10.0:1547251695.210776:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf0000] (24)-- 00000800:00000200:7.2:1547251695.210781:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:13.0:1547251695.210825:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)++ 00000800:00000200:13.0:1547251695.210829:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (24)-- 00000100:00000200:13.0:1547251695.210832:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b82444c500 x1622384012448512/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251702 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251695.210837:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad770 00000400:00000200:13.0:1547251695.210839:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251695.210841:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:15.0:1547251695.210843:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:13.0:1547251695.210845:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (23)-- 00000800:00000200:15.0:1547251695.210846:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:13.0:1547251695.210846:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)++ 00000800:00000200:13.0:1547251695.210847:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf0000] (23)-- 00000100:00000200:15.0:1547251695.210849:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b5a00 x1622384012448496/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251702 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:13.0:1547251695.210849:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960ed100 x1622384012448528/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251702 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251695.210854:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b630fe94c8 00000400:00000200:13.0:1547251695.210854:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914990 00000400:00000200:13.0:1547251695.210854:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:15.0:1547251695.210856:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251695.210856:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000400:00000200:15.0:1547251695.210858:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251695.210858:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (22)-- 00000800:00000200:13.0:1547251695.210859:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:13.0:1547251695.210861:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf0000] (21)-- 00000800:00000200:15.0:1547251695.210862:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:7.2:1547251695.210918:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251695.210924:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:13.0:1547251695.210931:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:13.0:1547251695.210934:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000800:00000200:15.0:1547251695.210937:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000400:00000200:13.0:1547251695.210937:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:15.0:1547251695.210939:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:13.0:1547251695.210941:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05300 00000400:00000200:15.0:1547251695.210943:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251695.210952:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x8014d [1] + 192 00000400:00000200:15.0:1547251695.210954:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c052f0 00000100:00000200:13.0:1547251695.210956:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b82444c500 x1622384012448512/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251702 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251695.210966:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:15.0:1547251695.210968:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x8015d [1] + 192 00000800:00000200:13.0:1547251695.210968:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000800:00000200:13.0:1547251695.210971:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000100:00000200:15.0:1547251695.210972:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b5a00 x1622384012448496/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251702 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251695.210972:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:13.0:1547251695.210974:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:13.0:1547251695.210976:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.93@o2ib10 00000400:00000200:13.0:1547251695.210978:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000400:00000200:13.0:1547251695.210981:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05310 00000400:00000200:15.0:1547251695.210982:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251695.210984:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)++ 00000100:00000200:5.0:1547251695.210985:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b82444c500 x1622384012448512/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251702 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251695.210987:0:12666:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (23)++ 00000800:00000200:15.0:1547251695.210989:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (24)-- 00000800:00000200:15.0:1547251695.210990:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:15.0:1547251695.210991:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:5.0:1547251695.210991:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b82444c500 x1622384012448512/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251702 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251695.210993:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000400:00000200:5.0:1547251695.210995:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad330 00000400:00000200:13.0:1547251695.210998:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x8016d [1] + 192 00010000:00080000:5.0:1547251695.210999:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b82444c500 x1622384012448512/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251702 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:9.2:1547251695.211000:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000100:00000200:2.0:1547251695.211001:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b5a00 x1622384012448496/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251702 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251695.211007:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000100:00000200:2.0:1547251695.211007:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b5a00 x1622384012448496/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251702 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251695.211008:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:15.0:1547251695.211009:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:15.0:1547251695.211010:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000400:00000200:2.0:1547251695.211011:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b630fe93b8 00000100:00000200:13.0:1547251695.211013:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960ed100 x1622384012448528/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251702 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:2.0:1547251695.211015:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b5a00 x1622384012448496/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251702 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:13.0:1547251695.211021:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251695.211022:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (21)++ 00000800:00000200:13.0:1547251695.211024:0:12665:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (22)++ 00000800:00000200:13.0:1547251695.211026:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:13.0:1547251695.211027:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000100:00000200:10.0:1547251695.211030:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960ed100 x1622384012448528/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251702 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:10.0:1547251695.211036:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960ed100 x1622384012448528/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251702 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:7.2:1547251695.211037:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000400:00000200:10.0:1547251695.211040:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914aa0 00010000:00080000:10.0:1547251695.211043:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba960ed100 x1622384012448528/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251702 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:15.0:1547251695.211049:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:15.0:1547251695.211054:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000800:00000200:15.0:1547251695.211055:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:15.0:1547251695.211056:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000100:00000200:15.0:1547251697.322675:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448544, portal 4 00000100:00000200:15.0:1547251697.322678:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448544, offset 0 00000400:00000200:15.0:1547251697.322681:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:15.0:1547251697.322684:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:15.0:1547251697.322687:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23110] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:15.0:1547251697.322690:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23110] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23110] 00000400:00000200:15.0:1547251697.322692:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:15.0:1547251697.322693:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251697.322695:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251697.322696:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9781, 9781] 00000400:00000200:15.0:1547251697.322698:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:15.0:1547251697.322702:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251697.322705:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:15.0:1547251697.322708:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251697.322709:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:15.0:1547251697.322710:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:15.0:1547251697.322713:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000100:00000200:3.0:1547251697.322729:0:12673:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448560, portal 4 00000100:00000200:3.0:1547251697.322732:0:12673:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448560, offset 0 00000400:00000200:3.0:1547251697.322737:0:12673:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:3.0:1547251697.322740:0:12673:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:3.0:1547251697.322742:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23111] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:3.0:1547251697.322745:0:12673:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23110] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23111] 00000100:00000200:9.0:1547251697.322747:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448576, portal 4 00000400:00000200:3.0:1547251697.322747:0:12673:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:3.0:1547251697.322749:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:9.0:1547251697.322750:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448576, offset 0 00000400:00000200:3.0:1547251697.322751:0:12673:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251697.322753:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:3.0:1547251697.322753:0:12673:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9781, 9782] 00000400:00000200:3.0:1547251697.322754:0:12673:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:9.0:1547251697.322756:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:9.0:1547251697.322758:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23111] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251697.322761:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23111] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23111] 00000800:00000200:5.2:1547251697.322763:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (21)++ 00000400:00000200:9.0:1547251697.322764:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:9.0:1547251697.322765:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251697.322767:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251697.322768:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 7], s:[9782, 9782] 00000400:00000200:9.0:1547251697.322770:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:9.0:1547251697.322774:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:10.0:1547251697.322775:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (22)++ 00000800:00000200:9.0:1547251697.322777:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000400:00000200:3.0:1547251697.322780:0:12673:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251697.322783:0:12673:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:10.0:1547251697.322784:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (23)-- 00000800:00000200:9.0:1547251697.322785:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251697.322786:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:3.0:1547251697.322786:0:12673:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:9.0:1547251697.322787:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000100:00000200:10.0:1547251697.322788:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8caaa00 x1622384012448544/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:3.0:1547251697.322788:0:12673:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (21)++ 00000800:00000200:9.0:1547251697.322790:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (23)-- 00000800:00000200:3.0:1547251697.322790:0:12673:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (23)++ 00000800:00000200:3.0:1547251697.322793:0:12673:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (23)-- 00000400:00000200:10.0:1547251697.322797:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb1330 00000400:00000200:10.0:1547251697.322798:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251697.322800:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000400:00000200:10.0:1547251697.322802:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251697.322805:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000800:00000200:7.2:1547251697.322838:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (22)++ 00000800:00000200:11.0:1547251697.322854:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:11.0:1547251697.322861:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000100:00000200:11.0:1547251697.322866:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf1b00 x1622384012448576/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251697.322867:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (23)++ 00000800:00000200:10.0:1547251697.322869:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (24)-- 00000400:00000200:11.0:1547251697.322872:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eabb0 00000100:00000200:10.0:1547251697.322872:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ad328df800 x1622384012448560/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251697.322874:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:11.0:1547251697.322877:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000400:00000200:10.0:1547251697.322878:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555ee0 00000400:00000200:10.0:1547251697.322879:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251697.322880:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (23)-- 00000800:00000200:11.0:1547251697.322881:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:10.0:1547251697.322882:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251697.322885:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:5.2:1547251697.322923:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (20)++ 00000800:00000200:7.2:1547251697.322928:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (20)++ 00000800:00000200:5.2:1547251697.322934:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:11.0:1547251697.322938:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (21)++ 00000800:00000200:11.0:1547251697.322941:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.91@o2ib10 00000800:00000200:10.0:1547251697.322941:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (21)++ 00000800:00000200:10.0:1547251697.322952:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[2] from 10.1.11.91@o2ib10 00000400:00000200:11.0:1547251697.322954:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:10.0:1547251697.322957:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251697.322959:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05320 00000400:00000200:10.0:1547251697.322961:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05330 00000400:00000200:11.0:1547251697.322963:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd78c9 [1] + 192 00000400:00000200:10.0:1547251697.322968:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd78d9 [1] + 192 00000100:00000200:11.0:1547251697.322972:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8caaa00 x1622384012448544/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251697.322982:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251697.322983:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ad328df800 x1622384012448560/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251697.322984:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669645400] (22)++ 00000800:00000200:11.0:1547251697.322986:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669645400] (23)-- 00000800:00000200:11.0:1547251697.322987:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:11.0:1547251697.322989:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251697.322991:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000400:00000200:10.0:1547251697.322994:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251697.322995:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97b048818800] (22)++ 00000400:00000200:11.0:1547251697.322996:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251697.322997:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97b048818800] (23)-- 00000800:00000200:10.0:1547251697.322998:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:11.0:1547251697.322999:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05340 00000800:00000200:10.0:1547251697.323000:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000400:00000200:11.0:1547251697.323002:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd78e9 [1] + 192 00000800:00000200:10.0:1547251697.323002:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:10.0:1547251697.323003:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000100:00000200:3.0:1547251697.323006:0:12673:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ad328df800 x1622384012448560/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251697.323012:0:12673:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ad328df800 x1622384012448560/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251697.323016:0:12673:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00000100:00000200:11.0:1547251697.323019:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf1b00 x1622384012448576/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:3.0:1547251697.323020:0:12673:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ad328df800 x1622384012448560/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:11.0:1547251697.323029:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:11.0:1547251697.323031:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251697.323032:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251697.323033:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00000100:00000200:9.0:1547251697.323038:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf1b00 x1622384012448576/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:9.0:1547251697.323044:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf1b00 x1622384012448576/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251697.323048:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea990 00010000:00080000:9.0:1547251697.323051:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf1b00 x1622384012448576/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:15.0:1547251697.323054:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8caaa00 x1622384012448544/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:15.0:1547251697.323060:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8caaa00 x1622384012448544/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251697.323065:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb15d8 00010000:00080000:15.0:1547251697.323068:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8caaa00 x1622384012448544/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251697.786683:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448592, portal 4 00000100:00000200:9.0:1547251697.786686:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448592, offset 0 00000400:00000200:9.0:1547251697.786689:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251697.786692:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:9.0:1547251697.786694:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23111] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:9.0:1547251697.786698:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23112] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23111] 00000400:00000200:9.0:1547251697.786701:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:9.0:1547251697.786702:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:9.0:1547251697.786703:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:14.0:1547251697.786704:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448624, portal 4 00000400:00000200:9.0:1547251697.786705:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9774, 9775] 00000400:00000200:9.0:1547251697.786706:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000100:00000200:14.0:1547251697.786707:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448624, offset 0 00000100:00000200:2.0:1547251697.786709:0:12670:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448608, portal 4 00000400:00000200:14.0:1547251697.786711:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:9.0:1547251697.786711:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000100:00000200:2.0:1547251697.786713:0:12670:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448608, offset 0 00000400:00000200:14.0:1547251697.786714:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:9.0:1547251697.786714:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000400:00000200:14.0:1547251697.786716:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23112] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:9.0:1547251697.786717:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4d80] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:2.0:1547251697.786717:0:12670:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:9.0:1547251697.786718:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ad57200000] (20)++ 00000400:00000200:14.0:1547251697.786719:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23112] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23112] 00000800:00000200:9.0:1547251697.786719:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57200000] (21)++ 00000400:00000200:2.0:1547251697.786720:0:12670:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:14.0:1547251697.786721:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:9.0:1547251697.786722:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ad57200000] (22)-- 00000400:00000200:14.0:1547251697.786723:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251697.786723:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23112] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:14.0:1547251697.786725:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251697.786726:0:12670:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23113] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23112] 00000400:00000200:14.0:1547251697.786727:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 8], s:[9775, 9775] 00000400:00000200:14.0:1547251697.786729:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000400:00000200:2.0:1547251697.786729:0:12670:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251697.786730:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251697.786732:0:12670:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:12.2:1547251697.786733:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (21)++ 00000400:00000200:2.0:1547251697.786733:0:12670:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[7, 7], s:[9775, 9776] 00000400:00000200:14.0:1547251697.786734:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:2.0:1547251697.786735:0:12670:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:14.0:1547251697.786737:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000800:00000200:14.0:1547251697.786740:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:2.0:1547251697.786740:0:12670:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251697.786741:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:14.0:1547251697.786742:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:2.0:1547251697.786743:0:12670:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:14.0:1547251697.786745:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:2.0:1547251697.786746:0:12670:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251697.786747:0:12670:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:2.0:1547251697.786748:0:12670:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:2.0:1547251697.786751:0:12670:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:0.2:1547251697.786759:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:0.2:1547251697.786766:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:5.0:1547251697.786794:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251697.786797:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57200000] (23)-- 00000800:00000200:4.0:1547251697.786798:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)++ 00000800:00000200:4.0:1547251697.786800:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badd4f5400] (23)-- 00000100:00000200:5.0:1547251697.786802:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf1e00 x1622384012448592/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251697.786803:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a5a00 x1622384012448624/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251697.786809:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea4c8 00000400:00000200:4.0:1547251697.786810:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:5.0:1547251697.786811:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251697.786811:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251697.786815:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:4.0:1547251697.786815:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251697.786818:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251697.786819:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251697.786820:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251697.786821:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:4.0:1547251697.786821:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251697.786823:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:5.0:1547251697.786824:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:4.0:1547251697.786824:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9b3b5700 x1622384012448608/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251697.786830:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b630fe9550 00000400:00000200:4.0:1547251697.786831:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251697.786834:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251697.786835:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:12.2:1547251697.786857:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:5.0:1547251697.786870:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251697.786879:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.1.11.92@o2ib10 00000800:00000200:5.0:1547251697.786882:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251697.786884:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251697.786885:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:4.0:1547251697.786886:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:12.2:1547251697.786898:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:0.2:1547251697.786904:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:5.0:1547251697.786911:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:12.2:1547251697.786915:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (20)++ 00000800:00000200:5.0:1547251697.786915:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000800:00000200:4.0:1547251697.786917:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)++ 00000400:00000200:5.0:1547251697.786919:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251697.786920:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251697.786924:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05350 00000400:00000200:4.0:1547251697.786924:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251697.786929:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4875 [1] + 192 00000400:00000200:4.0:1547251697.786929:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05360 00000400:00000200:4.0:1547251697.786933:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4895 [1] + 192 00000100:00000200:5.0:1547251697.786938:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf1e00 x1622384012448592/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251697.786960:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251697.786962:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (22)++ 00000100:00000200:4.0:1547251697.786962:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9b3b5700 x1622384012448608/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251697.786964:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (23)-- 00000800:00000200:5.0:1547251697.786965:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251697.786967:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000800:00000200:5.0:1547251697.786969:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251697.786972:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.92@o2ib10 00000400:00000200:5.0:1547251697.786976:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251697.786978:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251697.786979:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05370 00000800:00000200:4.0:1547251697.786980:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96a00] (22)++ 00000100:00000200:9.0:1547251697.786981:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf1e00 x1622384012448592/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251697.786982:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:4.0:1547251697.786983:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:4.0:1547251697.786984:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000100:00000200:9.0:1547251697.786986:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf1e00 x1622384012448592/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:4.0:1547251697.786986:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000400:00000200:9.0:1547251697.786990:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea990 00010000:00080000:9.0:1547251697.786994:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9ebf1e00 x1622384012448592/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251697.786994:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4885 [1] + 192 00000100:00000200:2.0:1547251697.786999:0:12670:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9b3b5700 x1622384012448608/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:2.0:1547251697.787004:0:12670:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9b3b5700 x1622384012448608/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:2.0:1547251697.787013:0:12670:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b630fe93b8 00000100:00000200:5.0:1547251697.787015:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a5a00 x1622384012448624/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:2.0:1547251697.787016:0:12670:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba9b3b5700 x1622384012448608/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:5.0:1547251697.787023:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251697.787025:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669647600] (21)++ 00000800:00000200:5.0:1547251697.787027:0:12662:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251697.787029:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669647600] (23)-- 00000800:00000200:5.0:1547251697.787029:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:12.2:1547251697.787040:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000800:00000200:4.0:1547251697.787054:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:4.0:1547251697.787059:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251697.787060:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251697.787063:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000100:00000200:14.0:1547251697.787071:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a5a00 x1622384012448624/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:14.0:1547251697.787077:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a5a00 x1622384012448624/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251697.787081:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00010000:00080000:14.0:1547251697.787085:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a5a00 x1622384012448624/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251704 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:9.0:1547251698.186686:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448640, portal 4 00000100:00000200:9.0:1547251698.186689:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448640, offset 0 00000400:00000200:9.0:1547251698.186692:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251698.186695:0:12679:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251698.186697:0:12679:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c05380 00000400:00000200:9.0:1547251698.186699:0:12679:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xd4f69 [1] + 0 00000400:00000200:9.0:1547251698.186702:0:12679:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97ba9d367ee0 00000100:00000200:9.0:1547251698.186703:0:12679:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000200:5.0:1547251698.186703:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448656, portal 4 00000100:00000200:5.0:1547251698.186706:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448656, offset 0 00000400:00000200:5.0:1547251698.186709:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:9.0:1547251698.186712:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba9d367ee0 00000400:00000200:5.0:1547251698.186712:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:9.0:1547251698.186713:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251698.186714:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c05390 00000100:00000200:9.0:1547251698.186717:0:12679:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba9ebf1800 x1622384012448640/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251705 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251698.186717:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe3115 [1] + 0 00000400:00000200:5.0:1547251698.186720:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97a75040b908 00000100:00000200:5.0:1547251698.186721:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:9.0:1547251698.186730:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245eaaa0 00000400:00000200:5.0:1547251698.186730:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97a75040b908 00000400:00000200:9.0:1547251698.186731:0:12679:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251698.186731:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:9.0:1547251698.186733:0:12679:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251698.186733:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b82444bc00 x1622384012448656/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251705 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251698.186734:0:13382:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012448640 00000400:00000200:5.0:1547251698.186738:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad220 00000400:00000200:5.0:1547251698.186739:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251698.186740:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00010000:00000200:8.0:1547251698.186743:0:13382:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c2df050 x1622384012448640/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:4/0 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:8.0:1547251698.186751:0:13382:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012448640, offset 192 00000400:00000200:8.0:1547251698.186754:0:13382:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:8.0:1547251698.186757:0:13382:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:8.0:1547251698.186759:0:13382:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c05380 00000400:00000200:8.0:1547251698.186762:0:13382:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf781 [1] + 192 00000100:00000200:7.0:1547251698.186762:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012448656 00000100:00000200:8.0:1547251698.186765:0:13382:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba9ebf1800 x1622384012448640/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251705 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00000200:7.0:1547251698.186771:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11bc50 x1622384012448656/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:4/0 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Interpret:/0/0 rc 0/0 00000400:00000200:8.0:1547251698.186773:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:8.0:1547251698.186775:0:13382:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688dd0 00000400:00000200:8.0:1547251698.186776:0:13382:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:7.0:1547251698.186778:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012448656, offset 192 00000400:00000200:8.0:1547251698.186779:0:13382:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000400:00000200:7.0:1547251698.186780:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000100:00000200:9.0:1547251698.186783:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba9ebf1800 x1622384012448640/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251705 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251698.186783:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000100:00000200:8.0:1547251698.186786:0:13382:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000400:00000200:7.0:1547251698.186786:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c05390 00000100:00000200:9.0:1547251698.186789:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba9ebf1800 x1622384012448640/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251705 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:9.0:1547251698.186794:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b8245ea990 00000400:00000200:7.0:1547251698.186796:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf791 [1] + 192 00010000:00080000:9.0:1547251698.186798:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97ba9ebf1800 x1622384012448640/t0(0) o13->cache1-OST0002-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251705 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:7.0:1547251698.186801:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b82444bc00 x1622384012448656/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251705 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251698.186809:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251698.186811:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eadd0 00000400:00000200:7.0:1547251698.186812:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251698.186813:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251698.186818:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251698.186818:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b82444bc00 x1622384012448656/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251705 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:5.0:1547251698.186823:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b82444bc00 x1622384012448656/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251705 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251698.186827:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad330 00010000:00080000:5.0:1547251698.186830:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b82444bc00 x1622384012448656/t0(0) o13->cache1-OST0000-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251705 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251698.202675:0:12677:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448672, portal 4 00000100:00000200:5.0:1547251698.202678:0:12677:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448672, offset 0 00000400:00000200:5.0:1547251698.202681:0:12677:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:5.0:1547251698.202684:0:12677:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:5.0:1547251698.202686:0:12677:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 224 into portal 7 MB=0x5c38cb6c053a0 00000400:00000200:5.0:1547251698.202689:0:12677:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 7 from 12345-0@lo of length 224/224 into md 0xe3145 [1] + 0 00000400:00000200:5.0:1547251698.202691:0:12677:0:(lib-md.c:65:lnet_md_unlink()) Queueing unlink of md ffff97b9073ad6e8 00000100:00000200:5.0:1547251698.202692:0:12677:0:(events.c:305:request_in_callback()) event type 2, status 0, service ost_create 00000400:00000200:5.0:1547251698.202700:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad6e8 00000400:00000200:5.0:1547251698.202701:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:5.0:1547251698.202704:0:12677:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b82444ef00 x1622384012448672/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251705 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251698.202708:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad220 00000400:00000200:5.0:1547251698.202709:0:12677:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251698.202710:0:12677:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:7.0:1547251698.202731:0:13183:0:(service.c:2145:ptlrpc_server_handle_request()) got req 1622384012448672 00010000:00000200:7.0:1547251698.202740:0:13183:0:(ldlm_lib.c:2946:target_send_reply_msg()) @@@ sending reply req@ffff97aa2c11b850 x1622384012448672/t0(0) o13->cache1-MDT0000-mdtlov_UUID@0@lo:4/0 lens 224/368 e 0 to 0 dl 1547251704 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000200:7.0:1547251698.202747:0:13183:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1622384012448672, offset 192 00000400:00000200:7.0:1547251698.202749:0:13183:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-0@lo 00000400:00000200:7.0:1547251698.202752:0:13183:0:(lib-move.c:4114:lnet_parse()) TRACE: 0@lo(0@lo) <- 0@lo : PUT - for me 00000400:00000200:7.0:1547251698.202755:0:13183:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-0@lo of length 336 into portal 4 MB=0x5c38cb6c053a0 00000400:00000200:7.0:1547251698.202757:0:13183:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-0@lo of length 336/336 into md 0xaf7a1 [1] + 192 00000100:00000200:7.0:1547251698.202761:0:13183:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b82444ef00 x1622384012448672/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251705 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:7.0:1547251698.202770:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:7.0:1547251698.202772:0:13183:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97bae17eaf68 00000400:00000200:7.0:1547251698.202773:0:13183:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:7.0:1547251698.202774:0:13183:0:(lib-msg.c:630:lnet_health_check()) health check: 0@lo->self: PUT: OK 00000100:00000200:5.0:1547251698.202778:0:12677:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b82444ef00 x1622384012448672/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251705 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:7.0:1547251698.202779:0:13183:0:(niobuf.c:962:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00000100:00000200:5.0:1547251698.202783:0:12677:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b82444ef00 x1622384012448672/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 1547251705 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251698.202787:0:12677:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b9073ad330 00010000:00080000:5.0:1547251698.202790:0:12677:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1126562) req@ffff97b82444ef00 x1622384012448672/t0(0) o13->cache1-OST0001-osc-MDT0000@0@lo:7/4 lens 224/336 e 0 to 0 dl 1547251705 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251700.218680:0:12671:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448688, portal 4 00000100:00000200:3.0:1547251700.218683:0:12671:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448688, offset 0 00000400:00000200:3.0:1547251700.218686:0:12671:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:3.0:1547251700.218689:0:12671:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:3.0:1547251700.218691:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23112] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:10.0:1547251700.218693:0:12681:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448720, portal 4 00000400:00000200:3.0:1547251700.218695:0:12671:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23114] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23112] 00000100:00000200:10.0:1547251700.218696:0:12681:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448720, offset 0 00000400:00000200:3.0:1547251700.218697:0:12671:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:3.0:1547251700.218698:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:10.0:1547251700.218699:0:12681:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000400:00000200:3.0:1547251700.218700:0:12671:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251700.218701:0:12671:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[8, 8], s:[9776, 9777] 00000400:00000200:10.0:1547251700.218702:0:12681:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000400:00000200:3.0:1547251700.218703:0:12671:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:10.0:1547251700.218705:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23113] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:10.0:1547251700.218708:0:12681:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23114] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23113] 00000400:00000200:3.0:1547251700.218708:0:12671:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:3.0:1547251700.218710:0:12671:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000400:00000200:10.0:1547251700.218711:0:12681:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:10.0:1547251700.218712:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000800:00000200:3.0:1547251700.218713:0:12671:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba9d3d0f00] -> 10.1.11.93@o2ib10 (2) version: 12 00000400:00000200:10.0:1547251700.218714:0:12681:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000100:00000200:6.0:1547251700.218714:0:12674:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448704, portal 4 00000400:00000200:10.0:1547251700.218715:0:12681:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 8], s:[9777, 9777] 00000800:00000200:3.0:1547251700.218716:0:12671:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (20)++ 00000400:00000200:10.0:1547251700.218717:0:12681:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.93@o2ib10 00000800:00000200:3.0:1547251700.218717:0:12671:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a5c2cf2800] (21)++ 00000100:00000200:6.0:1547251700.218718:0:12674:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448704, offset 0 00000800:00000200:3.0:1547251700.218720:0:12671:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a5c2cf2800] (22)-- 00000400:00000200:10.0:1547251700.218721:0:12681:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.93@o2ib10(10.0.11.93@o2ib10:10.0.11.93@o2ib10) : PUT try# 0 00000400:00000200:6.0:1547251700.218722:0:12674:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.93@o2ib10 00000800:00000200:10.0:1547251700.218724:0:12681:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.93@o2ib10 00000400:00000200:6.0:1547251700.218725:0:12674:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.93@o2ib10 local destination 00000800:00000200:10.0:1547251700.218727:0:12681:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447600] -> 10.0.11.93@o2ib10 (2) version: 12 00000800:00000200:10.0:1547251700.218729:0:12681:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669646200] (20)++ 00000400:00000200:6.0:1547251700.218729:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:254, d:10, s:23114] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:10.0:1547251700.218730:0:12681:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669646200] (21)++ 00000800:00000200:10.0:1547251700.218733:0:12681:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669646200] (22)-- 00000400:00000200:6.0:1547251700.218734:0:12674:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23114] with best_ni 10.0.11.90@o2ib10 [c:254, d:10, s:23114] 00000400:00000200:6.0:1547251700.218737:0:12674:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000800:00000200:7.2:1547251700.218738:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (21)++ 00000400:00000200:6.0:1547251700.218740:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251700.218742:0:12674:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:6.0:1547251700.218745:0:12674:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.93@o2ib10 c:[7, 7], s:[9777, 9778] 00000800:00000200:7.2:1547251700.218748:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (21)++ 00000400:00000200:6.0:1547251700.218748:0:12674:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.93@o2ib10 00000400:00000200:6.0:1547251700.218755:0:12674:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.93@o2ib10(10.0.11.93@o2ib10:10.1.11.93@o2ib10) : PUT try# 0 00000800:00000200:6.0:1547251700.218759:0:12674:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.93@o2ib10 00000800:00000200:6.0:1547251700.218763:0:12674:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641c00] -> 10.1.11.93@o2ib10 (2) version: 12 00000800:00000200:6.0:1547251700.218765:0:12674:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:6.0:1547251700.218766:0:12674:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:6.0:1547251700.218770:0:12674:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:9.2:1547251700.218787:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:15.0:1547251700.218794:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:15.0:1547251700.218797:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a5c2cf2800] (23)-- 00000100:00000200:15.0:1547251700.218800:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c0f00 x1622384012448688/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251707 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251700.218806:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555550 00000400:00000200:15.0:1547251700.218807:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:15.0:1547251700.218809:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:13.0:1547251700.218809:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (22)++ 00000800:00000200:15.0:1547251700.218812:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:13.0:1547251700.218812:0:12665:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669646200] (23)-- 00000800:00000200:15.0:1547251700.218814:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:15.0:1547251700.218815:0:12666:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ba9ab74200] (23)-- 00000100:00000200:13.0:1547251700.218816:0:12665:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ba960ea100 x1622384012448720/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251707 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:15.0:1547251700.218817:0:12666:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97ae5c437800 x1622384012448704/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251707 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251700.218821:0:12666:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990c38 00000400:00000200:13.0:1547251700.218821:0:12665:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914b28 00000400:00000200:15.0:1547251700.218822:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:13.0:1547251700.218822:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:15.0:1547251700.218824:0:12666:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.93@o2ib10: PUT: OK 00000800:00000200:15.0:1547251700.218825:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000400:00000200:13.0:1547251700.218825:0:12665:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.93@o2ib10: PUT: OK 00000800:00000200:15.0:1547251700.218827:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:13.0:1547251700.218828:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000800:00000200:15.0:1547251700.218829:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:13.0:1547251700.218830:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000800:00000200:7.2:1547251700.218847:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a5c2cf2800] (20)++ 00000800:00000200:15.0:1547251700.218860:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)++ 00000800:00000200:15.0:1547251700.218869:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d0[4] from 10.1.11.93@o2ib10 00000800:00000200:15.0:1547251700.218870:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (22)++ 00000800:00000200:15.0:1547251700.218871:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a5c2cf2800] (23)-- 00000800:00000200:15.0:1547251700.218872:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (22)-- 00000800:00000200:15.0:1547251700.218873:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a5c2cf2800] (21)-- 00000800:00000200:7.2:1547251700.218896:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669646200] (20)++ 00000800:00000200:9.2:1547251700.218901:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ba9ab74200] (20)++ 00000800:00000200:15.0:1547251700.218907:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669646200] (21)++ 00000800:00000200:15.0:1547251700.218909:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.0.11.93@o2ib10 00000400:00000200:15.0:1547251700.218913:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.0.11.93@o2ib10 : PUT - for me 00000800:00000200:13.0:1547251700.218913:0:12665:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:13.0:1547251700.218915:0:12665:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[1] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251700.218916:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c053d0 00000400:00000200:15.0:1547251700.218919:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x801ad [1] + 192 00000400:00000200:13.0:1547251700.218919:0:12665:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000100:00000200:15.0:1547251700.218923:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ba960ea100 x1622384012448720/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251707 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:13.0:1547251700.218923:0:12665:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c053b0 00000400:00000200:15.0:1547251700.218932:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:13.0:1547251700.218933:0:12665:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x8019d [1] + 192 00000800:00000200:15.0:1547251700.218934:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97a669646200] (22)++ 00000800:00000200:15.0:1547251700.218936:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97a669646200] (23)-- 00000800:00000200:15.0:1547251700.218936:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (22)-- 00000100:00000200:13.0:1547251700.218937:0:12665:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c0f00 x1622384012448688/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251707 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:15.0:1547251700.218938:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669646200] (21)-- 00000800:00000200:15.0:1547251700.218939:0:12666:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)++ 00000800:00000200:15.0:1547251700.218941:0:12666:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.93@o2ib10 00000400:00000200:15.0:1547251700.218951:0:12666:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.1.11.93@o2ib10 : PUT - for me 00000400:00000200:15.0:1547251700.218954:0:12666:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.93@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c053c0 00000400:00000200:13.0:1547251700.218955:0:12665:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:13.0:1547251700.218957:0:12665:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (23)++ 00000100:00000200:10.0:1547251700.218957:0:12681:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ba960ea100 x1622384012448720/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251707 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:13.0:1547251700.218960:0:12665:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (24)-- 00000800:00000200:13.0:1547251700.218961:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (23)-- 00000800:00000200:13.0:1547251700.218962:0:12665:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (22)-- 00000100:00000200:10.0:1547251700.218963:0:12681:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ba960ea100 x1622384012448720/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251707 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251700.218967:0:12681:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ae6a914aa0 00000400:00000200:15.0:1547251700.218970:0:12666:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.93@o2ib10 of length 336/336 into md 0x801bd [1] + 192 00010000:00080000:10.0:1547251700.218971:0:12681:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ba960ea100 x1622384012448720/t0(0) o13->cache1-OST000a-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251707 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:3.0:1547251700.218974:0:12671:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c0f00 x1622384012448688/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251707 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251700.218979:0:12671:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c0f00 x1622384012448688/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251707 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251700.218983:0:12671:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00000100:00000200:15.0:1547251700.218985:0:12666:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97ae5c437800 x1622384012448704/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251707 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00010000:00080000:3.0:1547251700.218987:0:12671:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c0f00 x1622384012448688/t0(0) o13->cache1-OST0009-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251707 ref 1 fl Rpc:R/0/0 rc 0/0 00000400:00000200:15.0:1547251700.218992:0:12666:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:15.0:1547251700.218994:0:12666:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ba9ab74200] (21)++ 00000800:00000200:15.0:1547251700.218995:0:12666:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ba9ab74200] (22)-- 00000800:00000200:15.0:1547251700.218996:0:12666:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ba9ab74200] (21)-- 00000100:00000200:6.0:1547251700.219003:0:12674:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97ae5c437800 x1622384012448704/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251707 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:6.0:1547251700.219009:0:12674:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97ae5c437800 x1622384012448704/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251707 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:6.0:1547251700.219013:0:12674:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b645990990 00010000:00080000:6.0:1547251700.219017:0:12674:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97ae5c437800 x1622384012448704/t0(0) o13->cache1-OST000b-osc-MDT0000@10.0.11.93@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251707 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:14.0:1547251702.330678:0:12683:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448736, portal 4 00000100:00000200:14.0:1547251702.330682:0:12683:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448736, offset 0 00000400:00000200:14.0:1547251702.330685:0:12683:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:14.0:1547251702.330687:0:12683:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:14.0:1547251702.330690:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23114] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000100:00000200:8.0:1547251702.330692:0:12680:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448752, portal 4 00000400:00000200:14.0:1547251702.330693:0:12683:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23115] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23114] 00000400:00000200:14.0:1547251702.330695:0:12683:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000100:00000200:8.0:1547251702.330695:0:12680:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448752, offset 0 00000400:00000200:14.0:1547251702.330696:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:14.0:1547251702.330698:0:12683:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251702.330698:0:12680:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:14.0:1547251702.330699:0:12683:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9783, 9782] 00000400:00000200:14.0:1547251702.330701:0:12683:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:8.0:1547251702.330701:0:12680:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:8.0:1547251702.330703:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23115] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251702.330707:0:12680:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23115] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23115] 00000400:00000200:8.0:1547251702.330709:0:12680:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251702.330711:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251702.330712:0:12680:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251702.330713:0:12680:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 8], s:[9783, 9783] 00000400:00000200:8.0:1547251702.330715:0:12680:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.91@o2ib10 00000400:00000200:8.0:1547251702.330720:0:12680:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:8.0:1547251702.330722:0:12680:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000400:00000200:14.0:1547251702.330725:0:12683:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.91@o2ib10(10.0.11.91@o2ib10:10.0.11.91@o2ib10) : PUT try# 0 00000800:00000200:14.0:1547251702.330728:0:12683:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.91@o2ib10 00000800:00000200:8.0:1547251702.330729:0:12680:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:14.0:1547251702.330731:0:12683:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447500] -> 10.0.11.91@o2ib10 (2) version: 12 00000800:00000200:8.0:1547251702.330731:0:12680:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (20)++ 00000800:00000200:8.0:1547251702.330732:0:12680:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (21)++ 00000800:00000200:14.0:1547251702.330733:0:12683:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669645400] (22)++ 00000800:00000200:14.0:1547251702.330735:0:12683:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669645400] (23)++ 00000800:00000200:8.0:1547251702.330735:0:12680:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (22)-- 00000800:00000200:14.0:1547251702.330738:0:12683:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669645400] (23)-- 00000100:00000200:2.0:1547251702.330747:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448768, portal 4 00000100:00000200:2.0:1547251702.330749:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448768, offset 0 00000400:00000200:2.0:1547251702.330752:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.91@o2ib10 00000400:00000200:2.0:1547251702.330755:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.91@o2ib10 local destination 00000400:00000200:2.0:1547251702.330758:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:254, d:10, s:23116] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:2.0:1547251702.330761:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23115] with best_ni 10.0.11.90@o2ib10 [c:254, d:10, s:23116] 00000400:00000200:2.0:1547251702.330763:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:2.0:1547251702.330765:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251702.330766:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:2.0:1547251702.330767:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.91@o2ib10 c:[8, 6], s:[9783, 9784] 00000400:00000200:2.0:1547251702.330769:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.91@o2ib10 00000400:00000200:2.0:1547251702.330774:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.91@o2ib10(10.0.11.91@o2ib10:10.1.11.91@o2ib10) : PUT try# 0 00000800:00000200:2.0:1547251702.330776:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.91@o2ib10 00000800:00000200:2.0:1547251702.330779:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97baa0641000] -> 10.1.11.91@o2ib10 (2) version: 12 00000800:00000200:2.0:1547251702.330780:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97b048818800] (20)++ 00000800:00000200:2.0:1547251702.330781:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97b048818800] (21)++ 00000800:00000200:2.0:1547251702.330784:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97b048818800] (22)-- 00000800:00000200:5.2:1547251702.330788:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669645400] (22)++ 00000800:00000200:10.0:1547251702.330804:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (23)++ 00000800:00000200:10.0:1547251702.330814:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (24)-- 00000100:00000200:10.0:1547251702.330818:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824754500 x1622384012448752/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:10.0:1547251702.330827:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688f68 00000400:00000200:10.0:1547251702.330828:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:11.0:1547251702.330829:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669645400] (23)++ 00000400:00000200:10.0:1547251702.330832:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251702.330833:0:12663:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669645400] (24)-- 00000800:00000200:7.2:1547251702.330834:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97b048818800] (21)++ 00000100:00000200:11.0:1547251702.330836:0:12663:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b81d4a2a00 x1622384012448736/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251702.330836:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (23)-- 00000800:00000200:10.0:1547251702.330838:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (22)-- 00000800:00000200:10.0:1547251702.330840:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97b048818800] (22)++ 00000800:00000200:10.0:1547251702.330841:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97b048818800] (23)-- 00000400:00000200:11.0:1547251702.330843:0:12663:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60000 00000400:00000200:11.0:1547251702.330844:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000200:10.0:1547251702.330844:0:12664:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b6268f3600 x1622384012448768/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251702.330847:0:12663:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.91@o2ib10: PUT: OK 00000800:00000200:11.0:1547251702.330850:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669645400] (21)-- 00000400:00000200:10.0:1547251702.330850:0:12664:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b630fe95d8 00000800:00000200:11.0:1547251702.330851:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (22)-- 00000400:00000200:10.0:1547251702.330851:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:10.0:1547251702.330854:0:12664:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.91@o2ib10: PUT: OK 00000800:00000200:10.0:1547251702.330856:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97b048818800] (21)-- 00000800:00000200:5.2:1547251702.330919:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (20)++ 00000800:00000200:7.2:1547251702.330925:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (20)++ 00000800:00000200:11.0:1547251702.330932:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (21)++ 00000800:00000200:11.0:1547251702.330937:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.91@o2ib10 00000800:00000200:10.0:1547251702.330939:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)++ 00000400:00000200:11.0:1547251702.330941:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.91@o2ib10 : PUT - for me 00000800:00000200:10.0:1547251702.330941:0:12664:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000400:00000200:10.0:1547251702.330954:0:12664:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251702.330955:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05400 00000400:00000200:10.0:1547251702.330958:0:12664:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c053f0 00000400:00000200:11.0:1547251702.330959:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7989 [1] + 192 00000400:00000200:10.0:1547251702.330963:0:12664:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7969 [1] + 192 00000100:00000200:11.0:1547251702.330968:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b6268f3600 x1622384012448768/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:11.0:1547251702.330980:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000100:00000200:10.0:1547251702.330981:0:12664:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824754500 x1622384012448752/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:11.0:1547251702.330982:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57202400] (22)++ 00000800:00000200:11.0:1547251702.330985:0:12663:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97ad57202400] (23)++ 00000800:00000200:11.0:1547251702.330987:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57202400] (24)-- 00000800:00000200:11.0:1547251702.330989:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (23)-- 00000800:00000200:11.0:1547251702.330990:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000800:00000200:11.0:1547251702.330992:0:12663:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:11.0:1547251702.330995:0:12663:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.0.11.91@o2ib10 00000800:00000200:5.2:1547251702.330998:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57202400] (21)++ 00000400:00000200:11.0:1547251702.330999:0:12663:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.91@o2ib10 : PUT - for me 00000400:00000200:11.0:1547251702.331002:0:12663:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.91@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c053e0 00000400:00000200:10.0:1547251702.331003:0:12664:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251702.331005:0:12664:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (23)++ 00000100:00000200:2.0:1547251702.331005:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b6268f3600 x1622384012448768/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251702.331007:0:12664:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96c00] (24)++ 00000800:00000200:10.0:1547251702.331009:0:12664:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (25)-- 00000800:00000200:10.0:1547251702.331010:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (24)-- 00000100:00000200:2.0:1547251702.331011:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b6268f3600 x1622384012448768/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251702.331012:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:10.0:1547251702.331014:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57202400] (22)++ 00000800:00000200:10.0:1547251702.331015:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97ad57202400] (23)-- 00000400:00000200:2.0:1547251702.331015:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b630fe93b8 00000800:00000200:10.0:1547251702.331016:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (22)-- 00000400:00000200:11.0:1547251702.331017:0:12663:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.91@o2ib10 of length 336/336 into md 0xd7959 [1] + 192 00000800:00000200:10.0:1547251702.331018:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57202400] (21)-- 00010000:00080000:2.0:1547251702.331018:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b6268f3600 x1622384012448768/t0(0) o13->cache1-OST0004-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/0 rc 0/0 00000800:00000200:7.2:1547251702.331020:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96c00] (22)++ 00000100:00000200:11.0:1547251702.331021:0:12663:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b81d4a2a00 x1622384012448736/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:10.0:1547251702.331028:0:12664:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)++ 00000800:00000200:10.0:1547251702.331030:0:12664:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96c00] (24)-- 00000400:00000200:11.0:1547251702.331031:0:12663:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:10.0:1547251702.331031:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (23)-- 00000800:00000200:11.0:1547251702.331032:0:12663:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)++ 00000800:00000200:10.0:1547251702.331032:0:12664:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251702.331034:0:12663:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badfa96c00] (22)-- 00000800:00000200:11.0:1547251702.331035:0:12663:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96c00] (21)-- 00000100:00000200:8.0:1547251702.331069:0:12680:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824754500 x1622384012448752/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251702.331075:0:12680:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824754500 x1622384012448752/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251702.331080:0:12680:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688660 00010000:00080000:8.0:1547251702.331084:0:12680:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824754500 x1622384012448752/t0(0) o13->cache1-OST0005-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:14.0:1547251702.331086:0:12683:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b81d4a2a00 x1622384012448736/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:14.0:1547251702.331092:0:12683:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b81d4a2a00 x1622384012448736/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:14.0:1547251702.331097:0:12683:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97adadf60d48 00010000:00080000:14.0:1547251702.331100:0:12683:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b81d4a2a00 x1622384012448736/t0(0) o13->cache1-OST0003-osc-MDT0000@10.0.11.91@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:8.0:1547251702.794677:0:12679:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448784, portal 4 00000100:00000200:8.0:1547251702.794679:0:12679:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448784, offset 0 00000400:00000200:8.0:1547251702.794682:0:12679:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:8.0:1547251702.794685:0:12679:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:8.0:1547251702.794687:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:256, d:10, s:23116] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000400:00000200:8.0:1547251702.794690:0:12679:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23116] with best_ni 10.0.11.90@o2ib10 [c:256, d:10, s:23116] 00000400:00000200:8.0:1547251702.794692:0:12679:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.0.11.90@o2ib10 00000400:00000200:8.0:1547251702.794694:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251702.794695:0:12679:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.0.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:8.0:1547251702.794697:0:12679:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 8], s:[9776, 9776] 00000100:00000200:15.0:1547251702.794698:0:12685:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448800, portal 4 00000400:00000200:8.0:1547251702.794698:0:12679:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.0.11.92@o2ib10 00000100:00000200:3.0:1547251702.794698:0:12672:0:(niobuf.c:884:ptl_send_rpc()) Setup reply buffer: 1024 bytes, xid 1622384012448816, portal 4 00000100:00000200:15.0:1547251702.794701:0:12685:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448800, offset 0 00000100:00000200:3.0:1547251702.794701:0:12672:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 7, xid 1622384012448816, offset 0 00000400:00000200:8.0:1547251702.794703:0:12679:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10:) -> 10.0.11.92@o2ib10(10.0.11.92@o2ib10:10.0.11.92@o2ib10) : PUT try# 0 00000400:00000200:15.0:1547251702.794704:0:12685:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000400:00000200:3.0:1547251702.794704:0:12672:0:(lib-move.c:4562:LNetPut()) LNetPut -> 12345-10.0.11.92@o2ib10 00000800:00000200:8.0:1547251702.794706:0:12679:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.0.11.92@o2ib10 00000400:00000200:15.0:1547251702.794707:0:12685:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000400:00000200:3.0:1547251702.794707:0:12672:0:(lib-move.c:2450:lnet_handle_send_case_locked()) Source ANY to MR: 10.0.11.92@o2ib10 local destination 00000800:00000200:8.0:1547251702.794708:0:12679:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97ba8c447480] -> 10.0.11.92@o2ib10 (2) version: 12 00000400:00000200:15.0:1547251702.794710:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23117] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:8.0:1547251702.794710:0:12679:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97a669647600] (20)++ 00000400:00000200:3.0:1547251702.794710:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.0.11.90@o2ib10 [c:255, d:10, s:23117] with best_ni not seleced [c:-2147483648, d:-1, s:0] 00000800:00000200:8.0:1547251702.794711:0:12679:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97a669647600] (21)++ 00000400:00000200:15.0:1547251702.794713:0:12685:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23116] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23117] 00000400:00000200:3.0:1547251702.794713:0:12672:0:(lib-move.c:1510:lnet_get_best_ni()) compare ni 10.1.11.90@o2ib10 [c:256, d:10, s:23116] with best_ni 10.0.11.90@o2ib10 [c:255, d:10, s:23117] 00000800:00000200:8.0:1547251702.794714:0:12679:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97a669647600] (22)-- 00000400:00000200:15.0:1547251702.794715:0:12685:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:3.0:1547251702.794715:0:12672:0:(lib-move.c:1553:lnet_get_best_ni()) selected best_ni 10.1.11.90@o2ib10 00000400:00000200:15.0:1547251702.794717:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251702.794717:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251702.794718:0:12685:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:3.0:1547251702.794718:0:12672:0:(lib-move.c:1766:lnet_select_peer_ni()) 10.1.11.90@o2ib10 ni_is_pref = 0 00000400:00000200:15.0:1547251702.794720:0:12685:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9776, 9777] 00000400:00000200:3.0:1547251702.794720:0:12672:0:(lib-move.c:1772:lnet_select_peer_ni()) 10.1.11.92@o2ib10 c:[8, 7], s:[9776, 9777] 00000400:00000200:15.0:1547251702.794722:0:12685:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000400:00000200:3.0:1547251702.794722:0:12672:0:(lib-move.c:1822:lnet_select_peer_ni()) sd_best_lpni = 10.1.11.92@o2ib10 00000800:00000200:12.2:1547251702.794725:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97a669647600] (21)++ 00000400:00000200:15.0:1547251702.794726:0:12685:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251702.794729:0:12685:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:15.0:1547251702.794732:0:12685:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000400:00000200:3.0:1547251702.794732:0:12672:0:(lib-move.c:1728:lnet_handle_send()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10:) -> 10.1.11.92@o2ib10(10.0.11.92@o2ib10:10.1.11.92@o2ib10) : PUT try# 0 00000800:00000200:15.0:1547251702.794734:0:12685:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (20)++ 00000800:00000200:15.0:1547251702.794735:0:12685:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (21)++ 00000800:00000200:3.0:1547251702.794735:0:12672:0:(o2iblnd_cb.c:1636:kiblnd_send()) sending 224 bytes in 1 frags to 12345-10.1.11.92@o2ib10 00000800:00000200:3.0:1547251702.794738:0:12672:0:(o2iblnd.c:403:kiblnd_find_peer_locked()) got peer_ni [ffff97b0a8ca4c00] -> 10.1.11.92@o2ib10 (2) version: 12 00000800:00000200:15.0:1547251702.794739:0:12685:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)-- 00000800:00000200:3.0:1547251702.794740:0:12672:0:(o2iblnd_cb.c:1516:kiblnd_launch_tx()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:3.0:1547251702.794741:0:12672:0:(o2iblnd_cb.c:1290:kiblnd_queue_tx_locked()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:3.0:1547251702.794743:0:12672:0:(o2iblnd_cb.c:1522:kiblnd_launch_tx()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:0.2:1547251702.794754:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251702.794784:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97a669647600] (22)++ 00000800:00000200:5.0:1547251702.794788:0:12662:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97a669647600] (23)-- 00000800:00000200:4.0:1547251702.794788:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)++ 00000800:00000200:4.0:1547251702.794790:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (24)-- 00000100:00000200:5.0:1547251702.794792:0:12662:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b824750f00 x1622384012448784/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000100:00000200:4.0:1547251702.794794:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97b0a8cacb00 x1622384012448800/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251702.794801:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb17f8 00000400:00000200:4.0:1547251702.794802:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:5.0:1547251702.794806:0:12662:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d6884c8 00000400:00000200:4.0:1547251702.794806:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000400:00000200:5.0:1547251702.794808:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:4.0:1547251702.794809:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (23)-- 00000400:00000200:5.0:1547251702.794811:0:12662:0:(lib-msg.c:630:lnet_health_check()) health check: 10.0.11.90@o2ib10->10.0.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251702.794811:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (22)-- 00000800:00000200:4.0:1547251702.794812:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)++ 00000800:00000200:5.0:1547251702.794813:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97a669647600] (21)-- 00000800:00000200:4.0:1547251702.794813:0:12661:0:(o2iblnd_cb.c:75:kiblnd_tx_done()) conn[ffff97badfa96a00] (23)-- 00000800:00000200:5.0:1547251702.794815:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (22)-- 00000100:00000200:4.0:1547251702.794816:0:12661:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff97a68b5c1200 x1622384012448816/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 2 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:4.0:1547251702.794822:0:12661:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555088 00000400:00000200:4.0:1547251702.794823:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:4.0:1547251702.794826:0:12661:0:(lib-msg.c:630:lnet_health_check()) health check: 10.1.11.90@o2ib10->10.1.11.92@o2ib10: PUT: OK 00000800:00000200:4.0:1547251702.794827:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badfa96a00] (21)-- 00000800:00000200:0.2:1547251702.794876:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97badd4f5400] (20)++ 00000800:00000200:12.2:1547251702.794881:0:0:0:(o2iblnd_cb.c:3710:kiblnd_cq_completion()) conn[ffff97ad57200000] (20)++ 00000800:00000200:5.0:1547251702.794891:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)++ 00000800:00000200:5.0:1547251702.794894:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[3] from 10.0.11.92@o2ib10 00000800:00000200:4.0:1547251702.794894:0:12661:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (21)++ 00000800:00000200:4.0:1547251702.794897:0:12661:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:5.0:1547251702.794899:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.1.11.90@o2ib10(10.1.11.90@o2ib10) <- 10.0.11.92@o2ib10 : PUT - for me 00000400:00000200:4.0:1547251702.794901:0:12661:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000400:00000200:5.0:1547251702.794904:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05410 00000400:00000200:4.0:1547251702.794905:0:12661:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05420 00000400:00000200:5.0:1547251702.794908:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4915 [1] + 192 00000400:00000200:4.0:1547251702.794912:0:12661:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4925 [1] + 192 00000100:00000200:5.0:1547251702.794917:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b824750f00 x1622384012448784/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251702.794929:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251702.794931:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97badd4f5400] (22)++ 00000100:00000200:4.0:1547251702.794931:0:12661:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97b0a8cacb00 x1622384012448800/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000800:00000200:5.0:1547251702.794933:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97badd4f5400] (23)-- 00000800:00000200:5.0:1547251702.794935:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (22)-- 00000800:00000200:5.0:1547251702.794936:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97badd4f5400] (21)-- 00000800:00000200:5.0:1547251702.794938:0:12662:0:(o2iblnd_cb.c:3832:kiblnd_scheduler()) conn[ffff97ad57200000] (22)++ 00000800:00000200:5.0:1547251702.794940:0:12662:0:(o2iblnd_cb.c:338:kiblnd_handle_rx()) Received d1[0] from 10.1.11.92@o2ib10 00000400:00000200:4.0:1547251702.794943:0:12661:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000400:00000200:5.0:1547251702.794953:0:12662:0:(lib-move.c:4114:lnet_parse()) TRACE: 10.0.11.90@o2ib10(10.0.11.90@o2ib10) <- 10.1.11.92@o2ib10 : PUT - for me 00000800:00000200:4.0:1547251702.794954:0:12661:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (23)++ 00000800:00000200:4.0:1547251702.794955:0:12661:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (24)-- 00000400:00000200:5.0:1547251702.794957:0:12662:0:(lib-ptl.c:571:lnet_ptl_match_md()) Request from 12345-10.0.11.92@o2ib10 of length 336 into portal 4 MB=0x5c38cb6c05430 00000800:00000200:4.0:1547251702.794957:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (23)-- 00000800:00000200:4.0:1547251702.794958:0:12661:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (22)-- 00000100:00000200:8.0:1547251702.794960:0:12679:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b824750f00 x1622384012448784/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:8.0:1547251702.794966:0:12679:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b824750f00 x1622384012448784/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:8.0:1547251702.794970:0:12679:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b81d688660 00000100:00000200:15.0:1547251702.794973:0:12685:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97b0a8cacb00 x1622384012448800/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00010000:00080000:8.0:1547251702.794974:0:12679:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b824750f00 x1622384012448784/t0(0) o13->cache1-OST0006-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:15.0:1547251702.794979:0:12685:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97b0a8cacb00 x1622384012448800/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:15.0:1547251702.794983:0:12685:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97ba8cbb15d8 00000400:00000200:5.0:1547251702.794986:0:12662:0:(lib-ptl.c:200:lnet_try_match_md()) Incoming put index 4 from 12345-10.0.11.92@o2ib10 of length 336/336 into md 0xe4935 [1] + 192 00010000:00080000:15.0:1547251702.794987:0:12685:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97b0a8cacb00 x1622384012448800/t0(0) o13->cache1-OST0007-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/336 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/0 rc 0/0 00000100:00000200:5.0:1547251702.794989:0:12662:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff97a68b5c1200 x1622384012448816/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000400:00000200:5.0:1547251702.794997:0:12662:0:(lib-msg.c:816:lnet_is_health_check()) health check = 0, status = 0, hstatus = 0 00000800:00000200:5.0:1547251702.794998:0:12662:0:(o2iblnd_cb.c:205:kiblnd_post_rx()) conn[ffff97ad57200000] (21)++ 00000800:00000200:5.0:1547251702.795000:0:12662:0:(o2iblnd_cb.c:234:kiblnd_post_rx()) conn[ffff97ad57200000] (22)-- 00000800:00000200:5.0:1547251702.795001:0:12662:0:(o2iblnd_cb.c:3848:kiblnd_scheduler()) conn[ffff97ad57200000] (21)-- 00000100:00000200:3.0:1547251702.795008:0:12672:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff97a68b5c1200 x1622384012448816/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000100:00000200:3.0:1547251702.795014:0:12672:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff97a68b5c1200 x1622384012448816/t0(0) o13->cache1-OST0008-osc-MDT0000@10.0.11.92@o2ib10:7/4 lens 224/368 e 0 to 0 dl 1547251709 ref 1 fl Rpc:R/0/ffffffff rc 0/-1 00000400:00000200:3.0:1547251702.795023:0:12672:0:(lib-md.c:69:lnet_md_unlink()) Unlinking md ffff97b824555a18 00010000:00080000:3.0:1547251702.795026:0:12672:0:(ldlm_request.c:1461:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 1502083) req@ffff97a68b5c1200 x1622384012448816/t