Dec 5 17:31:27 fe2 root: Start Testing Dec 5 17:31:50 fe2 kernel: Lustre: OBD class driver, http://wiki.whamcloud.com/ Dec 5 17:31:50 fe2 kernel: Lustre: Lustre Version: 1.8.6 Dec 5 17:31:50 fe2 kernel: Lustre: Build Version: jenkins-wc1-ga73a0cf-PRISTINE-2.6.32-71.el6.x86_64 Dec 5 17:31:50 fe2 kernel: Lustre: Listener bound to ib0:10.174.0.38:987:mlx4_0 Dec 5 17:31:50 fe2 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 Dec 5 17:31:50 fe2 kernel: Lustre: Added LNI 10.174.0.38@o2ib [8/64/0/180] Dec 5 17:31:50 fe2 kernel: Lustre: Listener bound to ib1:10.175.0.38:987:mlx4_0 Dec 5 17:31:50 fe2 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 Dec 5 17:31:50 fe2 kernel: Lustre: Added LNI 10.175.0.38@o2ib1 [8/64/0/180] Dec 5 17:31:50 fe2 kernel: Lustre: Listener bound to ib2:10.174.81.11:987:mlx4_1 Dec 5 17:31:50 fe2 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 Dec 5 17:31:50 fe2 kernel: Lustre: Added LNI 10.174.81.11@o2ib2 [8/64/0/180] Dec 5 17:31:50 fe2 kernel: Lustre: Lustre Client File System; http://www.lustre.org/ Dec 5 17:31:50 fe2 kernel: Lustre: MGC10.174.80.40@o2ib2: Reactivating import Dec 5 17:31:51 fe2 kernel: Lustre: Client scratch1-client has started Dec 5 17:32:09 fe2 kernel: Lustre: MGC10.174.80.42@o2ib2: Reactivating import Dec 5 17:32:14 fe2 kernel: Lustre: 20702:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1387377522115076 sent from scratch2-MDT0000-mdc-ffff880e747a1000 to NID 10.174.63.242@o2ib 5s ago has timed out (5s prior to deadline). Dec 5 17:32:14 fe2 kernel: req@ffff8815d9e30800 x1387377522115076/t0 o38->scratch2-MDT0000_UUID@10.174.63.242@o2ib:12/10 lens 368/584 e 0 to 1 dl 1323106334 ref 2 fl Rpc:N/0/0 rc 0/0 Dec 5 17:32:38 fe2 kernel: Lustre: 20702:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1387377522115257 sent from scratch2-MDT0000-mdc-ffff880e747a1000 to NID 10.174.63.252@o2ib 5s ago has timed out (5s prior to deadline). Dec 5 17:32:38 fe2 kernel: req@ffff8817bbc27400 x1387377522115257/t0 o38->scratch2-MDT0000_UUID@10.174.63.252@o2ib:12/10 lens 368/584 e 0 to 1 dl 1323106358 ref 2 fl Rpc:N/0/0 rc 0/0 Dec 5 17:32:44 fe2 kernel: Lustre: 20702:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1387377522115259 sent from scratch2-OST0000-osc-ffff880e747a1000 to NID 10.174.63.217@o2ib 5s ago has timed out (5s prior to deadline). Dec 5 17:32:44 fe2 kernel: req@ffff8815d9e30c00 x1387377522115259/t0 o8->scratch2-OST0000_UUID@10.174.63.217@o2ib:28/4 lens 368/584 e 0 to 1 dl 1323106364 ref 2 fl Rpc:N/0/0 rc 0/0 Dec 5 17:32:46 fe2 kernel: Lustre: 20727:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1387377522115480 sent from scratch2-MDT0000-mdc-ffff880e747a1000 to NID 10.175.31.242@o2ib 7s ago has timed out (7s prior to deadline). Dec 5 17:32:46 fe2 kernel: req@ffff8817b7679000 x1387377522115480/t0 o33->scratch2-MDT0000_UUID@10.175.31.242@o2ib:12/10 lens 408/600 e 0 to 1 dl 1323106366 ref 2 fl Rpc:/0/0 rc 0/0 Dec 5 17:32:46 fe2 kernel: Lustre: 20727:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 219 previous similar messages Dec 5 17:32:46 fe2 kernel: Lustre: scratch2-MDT0000-mdc-ffff880e747a1000: Connection to service scratch2-MDT0000 via nid 10.175.31.242@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 5 17:32:51 fe2 kernel: Lustre: 20702:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1387377522115482 sent from scratch2-OST0000-osc-ffff880e747a1000 to NID 10.174.63.218@o2ib 5s ago has timed out (5s prior to deadline). Dec 5 17:32:51 fe2 kernel: req@ffff881807d74400 x1387377522115482/t0 o8->scratch2-OST0000_UUID@10.174.63.218@o2ib:28/4 lens 368/584 e 0 to 1 dl 1323106371 ref 2 fl Rpc:N/0/0 rc 0/0 Dec 5 17:32:58 fe2 kernel: Lustre: 20702:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1387377522115703 sent from scratch2-OST0000-osc-ffff880e747a1000 to NID 10.175.31.217@o2ib 5s ago has timed out (5s prior to deadline). Dec 5 17:32:58 fe2 kernel: req@ffff8815e5989400 x1387377522115703/t0 o8->scratch2-OST0000_UUID@10.175.31.217@o2ib:28/4 lens 368/584 e 0 to 1 dl 1323106378 ref 2 fl Rpc:N/0/0 rc 0/0 Dec 5 17:32:58 fe2 kernel: Lustre: 20702:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 220 previous similar messages Dec 5 17:33:00 fe2 kernel: Lustre: 20703:0:(import.c:517:import_select_connection()) scratch2-OST0079-osc-ffff880e747a1000: tried all connections, increasing latency to 1s Dec 5 17:33:00 fe2 kernel: Lustre: 20703:0:(import.c:517:import_select_connection()) scratch2-OST007a-osc-ffff880e747a1000: tried all connections, increasing latency to 1s Dec 5 17:33:06 fe2 kernel: Lustre: 20703:0:(import.c:517:import_select_connection()) scratch2-OST0000-osc-ffff880e747a1000: tried all connections, increasing latency to 1s Dec 5 17:33:06 fe2 kernel: Lustre: 20703:0:(import.c:517:import_select_connection()) Skipped 9 previous similar messages Dec 5 17:33:07 fe2 kernel: Lustre: 20702:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1387377522116102 sent from scratch2-MDT0000-mdc-ffff880e747a1000 to NID 10.174.63.252@o2ib 7s ago has timed out (6s prior to deadline). Dec 5 17:33:07 fe2 kernel: req@ffff8815d9e30800 x1387377522116102/t0 o38->scratch2-MDT0000_UUID@10.174.63.252@o2ib:12/10 lens 368/584 e 0 to 1 dl 1323106386 ref 2 fl Rpc:N/0/0 rc 0/0 Dec 5 17:33:07 fe2 kernel: Lustre: 20702:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 429 previous similar messages Dec 5 17:33:15 fe2 kernel: Lustre: scratch1-MDT0000-mdc-ffff8817f3d6e400: Connection to service scratch1-MDT0000 via nid 10.174.31.241@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 5 17:33:16 fe2 kernel: Lustre: 20703:0:(import.c:517:import_select_connection()) scratch2-MDT0000-mdc-ffff880e747a1000: tried all connections, increasing latency to 2s Dec 5 17:33:16 fe2 kernel: Lustre: 20703:0:(import.c:517:import_select_connection()) Skipped 208 previous similar messages Dec 5 17:33:24 fe2 kernel: Lustre: 20703:0:(import.c:517:import_select_connection()) scratch2-OST0079-osc-ffff880e747a1000: tried all connections, increasing latency to 2s Dec 5 17:33:24 fe2 kernel: Lustre: 20702:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1387377522117330 sent from scratch2-MDT0000-mdc-ffff880e747a1000 to NID 10.174.63.242@o2ib 0s ago has failed due to network error (7s prior to deadline). Dec 5 17:33:24 fe2 kernel: req@ffff88178c93c000 x1387377522117330/t0 o38->scratch2-MDT0000_UUID@10.174.63.242@o2ib:12/10 lens 368/584 e 0 to 1 dl 1323106411 ref 1 fl Rpc:N/0/0 rc 0/0 Dec 5 17:33:24 fe2 kernel: Lustre: 20702:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 807 previous similar messages Dec 5 17:33:29 fe2 kernel: Lustre: 20703:0:(import.c:517:import_select_connection()) scratch1-MDT0000-mdc-ffff8817f3d6e400: tried all connections, increasing latency to 2s Dec 5 17:33:29 fe2 kernel: Lustre: 20703:0:(import.c:517:import_select_connection()) Skipped 10 previous similar messages Dec 5 17:33:37 fe2 kernel: Lustre: 20703:0:(import.c:517:import_select_connection()) scratch2-OST0000-osc-ffff880e747a1000: tried all connections, increasing latency to 2s Dec 5 17:33:37 fe2 kernel: Lustre: 20703:0:(import.c:517:import_select_connection()) Skipped 176 previous similar messages Dec 5 17:33:54 fe2 kernel: Lustre: 20703:0:(import.c:517:import_select_connection()) scratch2-OST0079-osc-ffff880e747a1000: tried all connections, increasing latency to 3s Dec 5 17:33:54 fe2 kernel: Lustre: 20703:0:(import.c:517:import_select_connection()) Skipped 386 previous similar messages Dec 5 17:33:56 fe2 kernel: Lustre: 20702:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1387377522118526 sent from scratch2-MDT0000-mdc-ffff880e747a1000 to NID 10.175.31.242@o2ib 8s ago has timed out (8s prior to deadline). Dec 5 17:33:56 fe2 kernel: req@ffff880f695b2800 x1387377522118526/t0 o38->scratch2-MDT0000_UUID@10.175.31.242@o2ib:12/10 lens 368/584 e 0 to 1 dl 1323106436 ref 2 fl Rpc:N/0/0 rc 0/0 Dec 5 17:33:56 fe2 kernel: Lustre: 20702:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 1579 previous similar messages Dec 5 17:34:19 fe2 kernel: LustreError: 11-0: an error occurred while communicating with 10.175.31.227@o2ib. The ost_connect operation failed with -19 Dec 5 17:34:19 fe2 kernel: LustreError: 11-0: an error occurred while communicating with 10.175.31.227@o2ib. The ost_connect operation failed with -19 Dec 5 17:34:24 fe2 kernel: Lustre: scratch1-MDT0000-mdc-ffff8817f3d6e400: Connection restored to service scratch1-MDT0000 using nid 10.174.31.241@o2ib. Dec 5 17:34:24 fe2 kernel: Lustre: scratch1-OST0000-osc-ffff8817f3d6e400: Connection restored to service scratch1-OST0000 using nid 10.174.31.201@o2ib. Dec 5 17:34:24 fe2 kernel: Lustre: Client scratch2-client has started Dec 5 17:34:41 fe2 root: End Testing