00010000:00010000:5.0:1392065759.472527:0:10020:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 979@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff881bf2728000/0xcf86cac758e2f8c0 lrc: 1/0,0 mode: --/PR res: 8628747995/115206 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x4e285e5bd84c2533 expref: 1102 pid: 4259 timeout: 7083106330 00010000:00010000:3.0:1392065759.685218:0:20661:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 979@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff881b3e213900/0xcf86cac758e34cdc lrc: 1/0,0 mode: --/PR res: 8628747995/115215 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x4e285e5bd84c25e9 expref: 1106 pid: 4259 timeout: 7083055542 00010000:00010000:9.0:1392065769.787513:0:22677:0:(ldlm_lock.c:626:ldlm_add_bl_work_item()) ### lock incompatible; sending blocking AST. ns: mdt-ffff881843d1a000 lock: ffff882e6e5ca6c0/0xcf86cac758f0aafc lrc: 2/0,0 mode: PR/PR res: 8625934250/15746 bits 0x3 rrc: 2 type: IBT flags: 0x4000000 remote: 0xfa2860e8477fa545 expref: 531 pid: 22677 timeout: 0 00010000:00010000:9.0:1392065769.787521:0:22677:0:(ldlm_lockd.c:809:ldlm_server_blocking_ast()) ### server preparing blocking AST ns: mdt-ffff881843d1a000 lock: ffff882e6e5ca6c0/0xcf86cac758f0aafc lrc: 3/0,0 mode: PR/PR res: 8625934250/15746 bits 0x3 rrc: 2 type: IBT flags: 0x4000020 remote: 0xfa2860e8477fa545 expref: 531 pid: 22677 timeout: 0 00010000:00010000:9.0:1392065769.787527:0:22677:0:(ldlm_lockd.c:468:ldlm_add_waiting_lock()) ### adding to wait list(timeout: 100, AT: on) ns: mdt-ffff881843d1a000 lock: ffff882e6e5ca6c0/0xcf86cac758f0aafc lrc: 4/0,0 mode: PR/PR res: 8625934250/15746 bits 0x3 rrc: 2 type: IBT flags: 0x4000020 remote: 0xfa2860e8477fa545 expref: 531 pid: 22677 timeout: 7083065645 00000100:00100000:9.0:1392065769.787532:0:22677:0:(client.c:1434:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc mdt_51:dc2-MDT0000_UUID:22677:1456766457060400:18@gni:104 00000100:00100000:0.0:1392065769.787543:0:11853:0:(events.c:300:request_in_callback()) peer: 12345-18@gni 00000100:00100000:1.0:1392065769.787550:0:29284:0:(service.c:1547:ptlrpc_server_handle_req_in()) got req x1459162827822891 00000100:00100000:9.0:1392065769.787553:0:22677:0:(client.c:2034:ptlrpc_set_wait()) set ffff882f3a6e64c0 going to sleep for 92 seconds 00010000:00010000:1.0:1392065769.787554:0:29284:0:(ldlm_lockd.c:2172:ldlm_cancel_hpreq_check()) ### hpreq cancel lock ns: mdt-ffff881843d1a000 lock: ffff882e6e5ca6c0/0xcf86cac758f0aafc lrc: 5/0,0 mode: PR/PR res: 8625934250/15746 bits 0x3 rrc: 2 type: IBT flags: 0x4000020 remote: 0xfa2860e8477fa545 expref: 532 pid: 22677 timeout: 7083065645 00000100:00100000:1.0:1392065769.787558:0:29284:0:(service.c:1290:ptlrpc_hpreq_reorder_nolock()) @@@ high priority req req@ffff88043be41800 x1459162827822891/t0(0) o103->e7c63df0-c694-b3c6-4c5e-31188a5e1fe7@18@gni:0/0 lens 296/0 e 0 to 0 dl 1392065861 ref 2 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:1.0:1392065769.787568:0:29284:0:(service.c:1724:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ldlm_cn_48:e7c63df0-c694-b3c6-4c5e-31188a5e1fe7+533:8269:x1459162827822891:12345-18@gni:103 00010000:00010000:1.0:1392065769.787571:0:29284:0:(ldlm_lockd.c:1433:ldlm_request_cancel()) ### server-side cancel handler START: 1 locks, starting at 0 00010000:00010000:1.0:1392065769.787574:0:29284:0:(ldlm_lockd.c:532:ldlm_del_waiting_lock()) ### removed ns: mdt-ffff881843d1a000 lock: ffff882e6e5ca6c0/0xcf86cac758f0aafc lrc: 4/0,0 mode: PR/PR res: 8625934250/15746 bits 0x3 rrc: 3 type: IBT flags: 0x4000020 remote: 0xfa2860e8477fa545 expref: 533 pid: 22677 timeout: 7083065645 00010000:00010000:1.0:1392065769.787577:0:29284:0:(ldlm_lockd.c:532:ldlm_del_waiting_lock()) ### wasn't waiting ns: mdt-ffff881843d1a000 lock: ffff882e6e5ca6c0/0xcf86cac758f0aafc lrc: 4/0,0 mode: PR/PR res: 8625934250/15746 bits 0x3 rrc: 3 type: IBT flags: 0x240000a0 remote: 0xfa2860e8477fa545 expref: 533 pid: 22677 timeout: 7083065645 00010000:00010000:1.0:1392065769.787582:0:29284:0:(ldlm_lock.c:644:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: mdt-ffff881843d1a000 lock: ffff88270b465b40/0xcf86cac758f0ab11 lrc: 3/0,1 mode: EX/EX res: 8625934250/15746 bits 0x7 rrc: 3 type: IBT flags: 0x5004000 remote: 0x0 expref: -99 pid: 22677 timeout: 0 00010000:00010000:1.0:1392065769.787587:0:29284:0:(ldlm_lockd.c:1468:ldlm_request_cancel()) ### server-side cancel handler END 00000100:00100000:0.0:1392065769.787598:0:11852:0:(events.c:300:request_in_callback()) peer: 12345-1032@gni 00000100:00100000:1.0:1392065769.787602:0:29284:0:(service.c:1771:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ldlm_cn_48:e7c63df0-c694-b3c6-4c5e-31188a5e1fe7+532:8269:x1459162827822891:12345-18@gni:103 Request procesed in 37us (59us total) trans 0 rc 0/0 00000100:00100000:3.0:1392065769.787607:0:23946:0:(service.c:1547:ptlrpc_server_handle_req_in()) got req x1459162943222603 00000100:00100000:3.0:1392065769.787616:0:23946:0:(service.c:1724:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc mdt_rdpg_110:5ef50be8-c61a-ebf4-e8d8-2605063698fe+3204:32650:x1459162943222603:12345-1032@gni:35 00000100:00100000:3.0:1392065769.787640:0:23946:0:(service.c:1771:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc mdt_rdpg_110:5ef50be8-c61a-ebf4-e8d8-2605063698fe+3203:32650:x1459162943222603:12345-1032@gni:35 Request procesed in 27us (42us total) trans 225878978647 rc 0/0 00010000:00010000:9.0:1392065769.787650:0:22677:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 18@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff882e6e5ca6c0/0xcf86cac758f0aafc lrc: 1/0,0 mode: --/PR res: 8625934250/15746 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0xfa2860e8477fa545 expref: 531 pid: 22677 timeout: 7083065645 00010000:00010000:9.0F:1392065769.787659:0:22677:0:(ldlm_lockd.c:532:ldlm_del_waiting_lock()) ### wasn't waiting ns: mdt-ffff881843d1a000 lock: ffff882e6e5ca6c0/0xcf86cac758f0aafc lrc: 1/0,0 mode: --/PR res: 8625934250/15746 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0xfa2860e8477fa545 expref: 531 pid: 22677 timeout: 7083065645 00010000:00010000:9.0:1392065769.787666:0:22677:0:(ldlm_lockd.c:532:ldlm_del_waiting_lock()) ### wasn't waiting ns: mdt-ffff881843d1a000 lock: ffff882e6e5ca6c0/0xcf86cac758f0aafc lrc: 1/0,0 mode: --/PR res: 8625934250/15746 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0xfa2860e8477fa545 expref: 531 pid: 22677 timeout: 7083065645 00010000:00010000:9.0:1392065769.787669:0:22677:0:(ldlm_lock.c:211:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: mdt-ffff881843d1a000 lock: ffff882e6e5ca6c0/0xcf86cac758f0aafc lrc: 0/0,0 mode: --/PR res: 8625934250/15746 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0xfa2860e8477fa545 expref: 531 pid: 22677 timeout: 7083065645 00000100:00100000:9.0:1392065769.787673:0:22677:0:(client.c:1773:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc mdt_51:dc2-MDT0000_UUID:22677:1456766457060400:18@gni:104 00010000:00010000:9.0:1392065769.787679:0:22677:0:(ldlm_request.c:435:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: mdt-ffff881843d1a000 lock: ffff88270b465b40/0xcf86cac758f0ab11 lrc: 3/0,1 mode: EX/EX res: 8625934250/15746 bits 0x7 rrc: 1 type: IBT flags: 0x4004000 remote: 0x0 expref: -99 pid: 22677 timeout: 0 00010000:00010000:4.0:1392065761.153446:0:28934:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 979@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff8817413b4900/0xcf86cac758e57ea3 lrc: 1/0,0 mode: --/PR res: 8628747995/115222 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x4e285e5bd84c2ea2 expref: 1096 pid: 28934 timeout: 7083057011 00010000:00010000:9.0:1392065769.787650:0:22677:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 18@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff882e6e5ca6c0/0xcf86cac758f0aafc lrc: 1/0,0 mode: --/PR res: 8625934250/15746 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0xfa2860e8477fa545 expref: 531 pid: 22677 timeout: 7083065645 00010000:00010000:5.0:1392065770.473506:0:4259:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 834@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff881b0f80a480/0xcf86cac758f1a02d lrc: 1/0,0 mode: --/PR res: 8626443370/32273 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0xf17ca59c73147e9a expref: 496 pid: 22677 timeout: 7083488331 00010000:00010000:5.0:1392065775.848779:0:28934:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 992@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff88190fd38900/0xcf86cac758f98f5a lrc: 1/0,0 mode: --/PR res: 8628562377/80800 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x854dd158852d2946 expref: 579 pid: 22677 timeout: 7083071706 00010000:00010000:5.0:1392065776.896670:0:28934:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 91@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff881d9558ad80/0xcf86cac758fafac0 lrc: 1/0,0 mode: --/PR res: 8625982410/65876 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x443da3d30e53cd49 expref: 484 pid: 28934 timeout: 7083072754 00010000:00010000:9.0:1392065792.049686:0:22677:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 979@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff882417a54000/0xcf86cac7590e59c2 lrc: 1/0,0 mode: --/PR res: 8628747995/115254 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x4e285e5bd84c820f expref: 1097 pid: 28934 timeout: 7083087907 00010000:00010000:7.0:1392065793.310551:0:4259:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 886@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff881e1159f240/0xcf86cac759102d02 lrc: 1/0,0 mode: --/PR res: 8629131887/7247 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x45c797795f03d49f expref: 736 pid: 22677 timeout: 7083089168 00010000:00010000:6.0:1392065793.753722:0:10020:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 91@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff8823a8a4f6c0/0xcf86cac75910b6fc lrc: 1/0,0 mode: --/PR res: 8625982410/65886 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x443da3d30e53e5c9 expref: 480 pid: 4259 timeout: 7083089611 00010000:00010000:7.0:1392065797.337077:0:4297:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 992@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff881cf55f6480/0xcf86cac75915a64b lrc: 1/0,0 mode: --/PR res: 8628562377/80818 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x854dd158852d5251 expref: 583 pid: 28934 timeout: 7083515194 00010000:00010000:0.0:1392065804.036019:0:10020:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 69@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff88207922ed80/0xcf86cac7591e8c00 lrc: 1/0,0 mode: --/PR res: 8628752123/69624 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x98a24d4d6899264e expref: 777 pid: 28934 timeout: 7083099893 00010000:00010000:2.0:1392065808.199034:0:28934:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 73@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff880c6d46a480/0xcf86cac75923fac9 lrc: 1/0,0 mode: --/PR res: 8628924112/97759 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x272e7ef58b488a0b expref: 1500 pid: 28934 timeout: 7083104056 00010000:00010000:6.0:1392065818.183114:0:4259:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 18@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff881233191240/0xcf86cac75932b58b lrc: 1/0,0 mode: --/PR res: 8625934250/20892 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0xfa2860e8479a21d6 expref: 642 pid: 4259 timeout: 7083114040 00010000:00010000:3.0:1392126988.544536:0:28664:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 881@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff88013d97d900/0xcf86cac8d01d5379 lrc: 1/0,0 mode: --/PR res: 8629199311/30641 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x611b61059f72a49a expref: 566 pid: 13956 timeout: 7144284402 00010000:00010000:7.0:1392126991.979012:0:22691:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 23@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff882eb830f000/0xcf86cac8d023d2c4 lrc: 1/0,0 mode: --/PR res: 8628490826/116013 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x39eb2131c6e0f2e9 expref: 594 pid: 13956 timeout: 7144287836 00010000:00010000:3.0:1392127002.022884:0:22696:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 832@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff8820d8de8240/0xcf86cac8d0325674 lrc: 1/0,0 mode: --/PR res: 8626440618/92970 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x591e9edc0d00763f expref: 522 pid: 22696 timeout: 7144297880 00010000:00010000:9.0:1392127008.715733:0:13956:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 870@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff882bc24d6900/0xcf86cac8d03a2d67 lrc: 1/0,0 mode: --/PR res: 8629313506/8094 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0xa6449e28f0c642ce expref: 618 pid: 4237 timeout: 7144304573 00010000:00010000:3.0:1392127011.458246:0:22696:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 71@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff882fb9ec5900/0xcf86cac8d03e90d2 lrc: 1/0,0 mode: --/PR res: 8625987914/98416 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x1e7e52281119351a expref: 578 pid: 22696 timeout: 7144307315 00010000:00010000:1.0:1392127014.124601:0:17162:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 65@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff88239e5f4d80/0xcf86cac8d0406a0f lrc: 1/0,0 mode: --/PR res: 8626410346/13094 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x362d90402b285531 expref: 422 pid: 17162 timeout: 7144309982 00010000:00010000:3.0:1392127029.846379:0:13956:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 832@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff880deb3be480/0xcf86cac8d0534721 lrc: 1/0,0 mode: --/PR res: 8626440618/92999 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0x591e9edc0d00b1a3 expref: 522 pid: 4237 timeout: 7144325704 00010000:00010000:5.0:1392127047.449980:0:22691:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 870@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff882d40bec480/0xcf86cac8d0676418 lrc: 1/0,0 mode: --/PR res: 8629313506/8125 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0xa6449e28f0c6858d expref: 622 pid: 22691 timeout: 7144343307 00010000:00010000:7.0:1392127053.898738:0:22696:0:(ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 989@gni) returned -22 from blocking AST - normal race ns: mdt-ffff881843d1a000 lock: ffff881c245e8000/0xcf86cac8d06cdf4b lrc: 1/0,0 mode: --/PR res: 8627903282/59087 bits 0x3 rrc: 2 type: IBT flags: 0x240000a0 remote: 0xf12ffa48c0c70ab expref: 488 pid: 22696 timeout: 7144349756