Aug 18 00:00:46 tscc-3-56 kernel: Lustre: 58839:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408345239/real 1408345239]  req@ffff880360f79c00 x1476519828087084/t0(0) o101->dolphin-MDT0000-mdc-ffff8808317cc800@172.25.32.25@tcp:12/10 lens 616/1136 e 0 to 1 dl 1408345246 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1
Aug 18 00:00:46 tscc-3-56 kernel: Lustre: 58839:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 00:00:46 tscc-3-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808317cc800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 00:00:47 tscc-1-11 kernel: Lustre: 45389:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408345240/real 1408345240]  req@ffff8805ea329000 x1476519740457488/t0(0) o35->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408345247 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 00:00:47 tscc-1-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026d94400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 00:00:52 tscc-0-31 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408345245/real 1408345245]  req@ffff8810349c9c00 x1476612857552452/t0(0) o400->dolphin-MDT0000-mdc-ffff880830a6b000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408345252 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:00:52 tscc-0-31 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 00:00:52 tscc-0-31 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830a6b000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 00:00:52 tscc-0-31 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:00:52 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408345246/real 1408345246]  req@ffff8804d3326000 x1476519828087144/t0(0) o38->dolphin-MDT0000-mdc-ffff8808317cc800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408345252 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:00:54 tscc-0-28 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408345247/real 1408345247]  req@ffff880824095000 x1476547206070532/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408345254 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:00:54 tscc-0-28 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 00:00:54 tscc-0-28 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:00:54 tscc-0-28 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831447800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 00:00:54 tscc-0-10 kernel: Lustre: 2761:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408345247/real 1408345247]  req@ffff880824b6fc00 x1476547227031188/t0(0) o400->dolphin-MDT0000-mdc-ffff8810322ef000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408345254 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:00:54 tscc-0-10 kernel: Lustre: 2761:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 00:00:54 tscc-0-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810322ef000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 00:00:54 tscc-0-10 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:00:55 tscc-1-11 kernel: Lustre: 2754:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408345244/real 1408345244]  req@ffff88101416e800 x1476519740457532/t0(0) o103->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.25@tcp:17/18 lens 328/224 e 0 to 1 dl 1408345255 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 00:00:55 tscc-1-48 kernel: Lustre: 2735:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408345248/real 1408345248]  req@ffff88101e8eb800 x1476537001036560/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408345255 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:00:55 tscc-1-48 kernel: Lustre: 2735:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 00:00:55 tscc-1-48 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:00:55 tscc-1-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028f17800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 00:00:55 tscc-0-57 kernel: Lustre: 2240:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408345248/real 1408345248]  req@ffff880ff8a26c00 x1475562021249684/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408345255 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:00:55 tscc-0-57 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:00:55 tscc-0-57 kernel: Lustre: dolphin-MDT0000-mdc-ffff881032828c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 00:00:57 tscc-1-1 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408345250/real 1408345250]  req@ffff880b48547400 x1476519529545248/t0(0) o400->dolphin-MDT0000-mdc-ffff882026059c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408345257 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:00:57 tscc-1-1 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 00:00:57 tscc-1-1 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:00:57 tscc-1-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026059c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 00:00:57 tscc-0-28 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831447800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 00:00:57 tscc-0-28 kernel: Lustre: Skipped 1 previous similar message
Aug 18 00:00:57 tscc-0-31 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830a6b000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 00:00:57 tscc-0-31 kernel: Lustre: Skipped 1 previous similar message
Aug 18 00:00:57 tscc-0-57 kernel: Lustre: dolphin-MDT0000-mdc-ffff881032828c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 00:00:57 tscc-1-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026059c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 00:00:57 tscc-1-1 kernel: Lustre: Skipped 1 previous similar message
Aug 18 00:00:57 tscc-1-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028f17800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 00:00:57 tscc-1-48 kernel: Lustre: Skipped 1 previous similar message
Aug 18 00:00:57 tscc-0-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810322ef000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 00:00:57 tscc-0-10 kernel: Lustre: Skipped 1 previous similar message
Aug 18 00:01:17 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408345271/real 1408345271]  req@ffff88083063ac00 x1476519828087196/t0(0) o38->dolphin-MDT0000-mdc-ffff8808317cc800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408345277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:01:22 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408345272/real 1408345272]  req@ffff8802c79b3800 x1476519740457644/t0(0) o38->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408345282 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:01:22 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 00:01:36 tscc-3-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808317cc800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 00:01:36 tscc-3-56 kernel: Lustre: Skipped 1 previous similar message
Aug 18 00:01:37 tscc-1-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026d94400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 00:21:25 tscc-0-31 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408346477/real 1408346477]  req@ffff8803d833c000 x1476612857565424/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408346485 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:21:25 tscc-0-31 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 00:21:25 tscc-0-31 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:21:25 tscc-gpu-5-4 kernel: Lustre: 2630:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408346476/real 1408346476]  req@ffff88045fe0e400 x1476612337325536/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408346485 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:21:25 tscc-gpu-5-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:21:28 tscc-0-28 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408346479/real 1408346479]  req@ffff88082f45ac00 x1476547206083508/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408346488 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:21:28 tscc-0-28 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 00:21:28 tscc-0-28 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:21:28 tscc-0-10 kernel: Lustre: 2761:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408346479/real 1408346479]  req@ffff880826789000 x1476547227044160/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408346488 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:21:28 tscc-0-10 kernel: Lustre: 2761:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 00:21:28 tscc-0-10 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:21:29 tscc-1-48 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408346480/real 1408346480]  req@ffff881fdd13f800 x1476537001049536/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408346489 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:21:29 tscc-1-48 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 00:21:29 tscc-1-48 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:21:29 tscc-0-28 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 00:21:29 tscc-0-28 kernel: Lustre: Skipped 1 previous similar message
Aug 18 00:21:29 tscc-0-31 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 00:21:29 tscc-0-31 kernel: Lustre: Skipped 1 previous similar message
Aug 18 00:21:29 tscc-1-48 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 00:21:29 tscc-1-48 kernel: Lustre: Skipped 1 previous similar message
Aug 18 00:21:29 tscc-gpu-5-4 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 00:21:29 tscc-0-10 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 00:21:29 tscc-0-10 kernel: Lustre: Skipped 1 previous similar message
Aug 18 00:47:10 tscc-1-56 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408348008/real 1408348008]  req@ffff88202f940c00 x1476536955975264/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408348030 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:47:10 tscc-1-56 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:47:10 tscc-1-56 kernel: LustreError: 11-0: MGC172.25.32.53@tcp: Communicating with 172.25.32.25@tcp, operation mgs_connect failed with -16.
Aug 18 00:47:35 tscc-1-56 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 00:58:04 tscc-1-6 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408348668/real 1408348668]  req@ffff881bb1eea400 x1476519509787048/t0(0) o103->dolphin-MDT0000-mdc-ffff8810266d7400@172.25.32.25@tcp:17/18 lens 968/224 e 0 to 1 dl 1408348684 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 00:58:04 tscc-1-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810266d7400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 00:58:08 tscc-1-11 kernel: Lustre: 2756:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408348662/real 1408348662]  req@ffff880e295a0800 x1476519740646916/t0(0) o103->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.25@tcp:17/18 lens 328/224 e 0 to 1 dl 1408348688 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 00:58:08 tscc-1-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026d94400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 00:58:11 tscc-1-56 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408348655/real 1408348655]  req@ffff881fffc19800 x1476536955981900/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408348691 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:58:11 tscc-1-56 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:58:13 tscc-0-7 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408348659/real 1408348659]  req@ffff88081b834800 x1476547271015792/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408348693 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:58:13 tscc-0-7 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:58:13 tscc-1-35 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408348656/real 1408348656]  req@ffff88112bbc2c00 x1476285713002356/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408348693 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:58:13 tscc-1-35 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 00:58:13 tscc-1-35 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:58:13 tscc-1-53 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408348656/real 1408348656]  req@ffff8810267ea000 x1476537035687972/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408348693 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:58:13 tscc-1-53 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 00:58:13 tscc-1-53 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:58:13 tscc-0-62 kernel: Lustre: 2459:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408348658/real 1408348658]  req@ffff880756a3cc00 x1475809553171688/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408348693 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:58:13 tscc-0-62 kernel: Lustre: 2459:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 00:58:13 tscc-0-62 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:58:13 tscc-2-52 kernel: Lustre: 2970:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408348656/real 1408348656]  req@ffff88067d15e000 x1476290264984000/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408348693 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:58:13 tscc-2-52 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:58:13 tscc-2-10 kernel: Lustre: 2969:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408348658/real 1408348658]  req@ffff8803abe60c00 x1476519077782124/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408348693 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 00:58:13 tscc-2-10 kernel: Lustre: 2969:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 00:58:13 tscc-2-10 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 00:58:13 tscc-1-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810266d7400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 00:58:13 tscc-1-53 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 00:58:13 tscc-1-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026d94400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 00:58:13 tscc-0-7 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 00:58:13 tscc-1-56 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 00:58:13 tscc-0-62 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 00:58:13 tscc-0-62 kernel: Lustre: Skipped 1 previous similar message
Aug 18 00:58:13 tscc-1-35 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 00:58:13 tscc-1-35 kernel: Lustre: Skipped 1 previous similar message
Aug 18 00:58:13 tscc-2-10 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 00:58:13 tscc-2-10 kernel: Lustre: Skipped 1 previous similar message
Aug 18 00:58:13 tscc-2-52 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 01:26:38 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 01:26:38 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 01:26:38 tscc-login1 kernel: LustreError: 49112:0:(file.c:171:ll_close_inode_openhandle()) inode 144115537578819586 mdc close failed: rc = -108
Aug 18 01:26:38 tscc-login1 kernel: LustreError: 49112:0:(file.c:171:ll_close_inode_openhandle()) Skipped 3 previous similar messages
Aug 18 01:26:38 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 03:43:41 tscc-1-11 kernel: Lustre: 2753:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408358613/real 1408358613]  req@ffff8819c1288400 x1476519741286020/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408358621 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 03:43:41 tscc-1-11 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 03:43:42 tscc-1-11 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 04:02:04 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 04:02:04 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 04:02:04 tscc-login1 kernel: LustreError: Skipped 4 previous similar messages
Aug 18 04:02:04 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 04:02:04 tscc-login1 kernel: LustreError: 4049:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 04:02:04 tscc-login1 kernel: LustreError: 4049:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 165 previous similar messages
Aug 18 04:02:04 tscc-login1 kernel: LustreError: 4049:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID [0x14e00001:0xa0252bbc:0x0] error: rc = -108
Aug 18 04:02:04 tscc-login1 kernel: LustreError: 4049:0:(file.c:171:ll_close_inode_openhandle()) inode 144115535431335952 mdc close failed: rc = -108
Aug 18 04:02:04 tscc-login1 kernel: LustreError: 4049:0:(file.c:171:ll_close_inode_openhandle()) Skipped 9 previous similar messages
Aug 18 04:02:04 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 04:34:34 tscc-0-59 kernel: LustreError: 11-0: dolphin-OST0022-osc-ffff880832bf5400: Communicating with 172.25.33.115@tcp, operation ldlm_enqueue failed with -12.
Aug 18 04:59:06 tscc-3-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808317cc800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 04:59:06 tscc-3-56 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8808317cc800: Communicating with 172.25.32.25@tcp, operation mds_close failed with -107.
Aug 18 04:59:06 tscc-3-56 kernel: LustreError: Skipped 1 previous similar message
Aug 18 04:59:06 tscc-3-56 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8808317cc800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 04:59:06 tscc-3-56 kernel: LustreError: 60218:0:(file.c:171:ll_close_inode_openhandle()) inode 144115512782139511 mdc close failed: rc = -5
Aug 18 04:59:06 tscc-3-56 kernel: LustreError: 60222:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 04:59:06 tscc-3-56 kernel: LustreError: 60222:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 3 previous similar messages
Aug 18 04:59:06 tscc-3-56 kernel: LustreError: 60218:0:(file.c:171:ll_close_inode_openhandle()) Skipped 39 previous similar messages
Aug 18 04:59:06 tscc-3-56 kernel: LustreError: 60219:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200002f13:0x9ed5:0x0] error -108.
Aug 18 04:59:06 tscc-3-56 kernel: LustreError: 60219:0:(vvp_io.c:1229:vvp_io_init()) Skipped 4 previous similar messages
Aug 18 04:59:06 tscc-3-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808317cc800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 05:09:28 tscc-0-60 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408363760/real 1408363760]  req@ffff8810321b1400 x1476700762424564/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408363768 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:09:28 tscc-0-60 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:09:28 tscc-0-60 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:11:25 tscc-1-4 kernel: Lustre: 2743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408363875/real 1408363875]  req@ffff880bc7b51800 x1475918801531884/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408363885 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:11:25 tscc-1-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:11:25 tscc-2-55 kernel: Lustre: 2971:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408363873/real 1408363873]  req@ffff880526aae000 x1476519001557332/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408363885 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:11:25 tscc-2-55 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:11:25 tscc-1-46 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408363873/real 1408363873]  req@ffff880f5bf70c00 x1476536973022536/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408363885 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:11:25 tscc-1-46 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:11:29 tscc-1-4 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:11:36 tscc-2-55 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408363885/real 1408363885]  req@ffff880c74242000 x1476519001557552/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408363896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:11:36 tscc-1-46 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408363885/real 1408363885]  req@ffff880efa048800 x1476536973022760/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408363896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:11:41 tscc-1-54 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408363888/real 1408363888]  req@ffff881f230d2800 x1476536987686440/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408363901 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:11:41 tscc-1-54 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:11:53 tscc-1-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408363901/real 1408363901]  req@ffff882006860c00 x1476536987686708/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408363913 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:12:01 tscc-2-55 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408363910/real 1408363910]  req@ffff88055f4e6000 x1476519001557632/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408363921 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:12:01 tscc-1-46 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408363910/real 1408363910]  req@ffff880efa136400 x1476536973022840/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408363921 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:12:15 tscc-2-55 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:12:15 tscc-1-46 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:12:18 tscc-1-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408363926/real 1408363926]  req@ffff881dba4a3c00 x1476536987687264/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408363938 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:12:32 tscc-1-54 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:20:37 tscc-1-59 kernel: Lustre: 2744:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408364414/real 1408364414]  req@ffff880e400e4000 x1476536863324304/t0(0) o103->dolphin-MDT0000-mdc-ffff880831706400@172.25.32.25@tcp:17/18 lens 328/224 e 0 to 1 dl 1408364437 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 05:20:37 tscc-1-59 kernel: Lustre: 2744:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 05:20:37 tscc-1-59 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831706400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 05:20:40 tscc-2-55 kernel: Lustre: 2966:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408364414/real 1408364414]  req@ffff881054c9cc00 x1476519001587288/t0(0) o103->dolphin-MDT0000-mdc-ffff882068dc4c00@172.25.32.25@tcp:17/18 lens 328/224 e 0 to 1 dl 1408364440 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 05:20:40 tscc-2-55 kernel: Lustre: dolphin-MDT0000-mdc-ffff882068dc4c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 05:20:43 tscc-1-33 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408364419/real 1408364419]  req@ffff880e32fe5800 x1476536972002688/t0(0) o103->dolphin-MDT0000-mdc-ffff8810264d1400@172.25.32.25@tcp:17/18 lens 328/224 e 0 to 1 dl 1408364443 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 05:20:43 tscc-1-33 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810264d1400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 05:20:44 tscc-1-46 kernel: Lustre: 35750:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408364414/real 1408364414]  req@ffff881d4f09d400 x1476536973062460/t0(0) o35->dolphin-MDT0000-mdc-ffff88101d826800@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408364444 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 05:20:44 tscc-1-46 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101d826800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 05:20:46 tscc-1-41 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408364424/real 1408364424]  req@ffff881dc720b400 x1476537045402160/t0(0) o103->dolphin-MDT0000-mdc-ffff882026991400@172.25.32.25@tcp:17/18 lens 328/224 e 0 to 1 dl 1408364446 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 05:20:46 tscc-1-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026991400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 05:20:48 tscc-1-46 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101d826800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 05:20:48 tscc-1-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026991400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 05:20:48 tscc-2-55 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882068dc4c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 05:20:48 tscc-1-59 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831706400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 05:20:48 tscc-1-59 kernel: Lustre: Skipped 1 previous similar message
Aug 18 05:20:48 tscc-1-33 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810264d1400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 05:21:06 tscc-2-55 kernel: Lustre: dolphin-MDT0000-mdc-ffff882068dc4c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 05:22:01 tscc-3-41 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408364505/real 1408364505]  req@ffff881e7c840400 x1476519855665160/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408364521 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:22:01 tscc-3-41 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:22:02 tscc-3-41 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:24:32 tscc-0-15 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408364656/real 1408364656]  req@ffff8803f23cc000 x1476612876082988/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408364672 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:24:32 tscc-0-15 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 05:24:32 tscc-0-15 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:24:33 tscc-0-31 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408364660/real 1408364660]  req@ffff88035d02fc00 x1476612857757508/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408364673 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:24:33 tscc-0-31 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:24:33 tscc-0-28 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408364663/real 1408364663]  req@ffff880829f47400 x1476547206275472/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408364673 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:24:33 tscc-0-28 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:24:33 tscc-3-51 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408364661/real 1408364661]  req@ffff88100a9dcc00 x1476519881409356/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408364673 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:24:33 tscc-3-51 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 05:24:33 tscc-3-51 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:24:33 tscc-1-6 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408364659/real 1408364659]  req@ffff88031bd43c00 x1476519511051740/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408364673 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:24:33 tscc-1-6 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:24:33 tscc-0-36 kernel: Lustre: 2743:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408364660/real 1408364660]  req@ffff880228a19000 x1476547204557168/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408364673 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:24:33 tscc-0-36 kernel: Lustre: 2743:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 05:24:33 tscc-0-36 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:24:33 tscc-0-15 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:24:33 tscc-0-15 kernel: Lustre: Skipped 1 previous similar message
Aug 18 05:24:33 tscc-0-31 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:24:33 tscc-0-36 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:24:33 tscc-0-28 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:24:33 tscc-3-51 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:24:33 tscc-3-51 kernel: Lustre: Skipped 1 previous similar message
Aug 18 05:24:33 tscc-1-6 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:39:52 tscc-0-5 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408365585/real 1408365585]  req@ffff8808314a2c00 x1476638712099268/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408365592 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:39:52 tscc-0-5 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:39:54 tscc-1-59 kernel: Lustre: 2738:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408365587/real 1408365587]  req@ffff8804b4011800 x1476536863400584/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408365594 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 05:39:54 tscc-1-59 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 05:39:55 tscc-0-5 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:39:55 tscc-1-59 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 05:47:50 tscc-1-41 kernel: Lustre: 49589:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408366061/real 1408366061]  req@ffff880de6870c00 x1476537045518900/t0(0) o35->dolphin-MDT0000-mdc-ffff882026991400@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408366070 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 05:47:50 tscc-1-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026991400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 05:47:50 tscc-1-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026991400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 06:28:17 tscc-1-32 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408368487/real 1408368487]  req@ffff880dc1367800 x1476439897811044/t0(0) o103->dolphin-MDT0000-mdc-ffff8810258fb000@172.25.32.25@tcp:17/18 lens 328/224 e 0 to 1 dl 1408368497 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 06:28:17 tscc-1-32 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810258fb000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 06:28:20 tscc-1-32 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810258fb000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 07:47:16 tscc-2-48 kernel: Lustre: 32055:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408373159/real 1408373159]  req@ffff8806ed30ac00 x1476518979401844/t0(0) o35->dolphin-MDT0000-mdc-ffff882066596000@172.25.32.25@tcp:23/10 lens 392/4328 e 2 to 1 dl 1408373236 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 07:47:16 tscc-2-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff882066596000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 07:47:16 tscc-2-48 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882066596000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:47:19 tscc-2-51 kernel: Lustre: 30517:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408373148/real 1408373148]  req@ffff881f76132000 x1476518959487312/t0(0) o35->dolphin-MDT0000-mdc-ffff881067285c00@172.25.32.25@tcp:23/10 lens 392/4328 e 1 to 1 dl 1408373239 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 07:47:19 tscc-2-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff881067285c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 07:47:19 tscc-2-51 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881067285c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:47:23 tscc-1-8 kernel: Lustre: 9619:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408373152/real 1408373152]  req@ffff883fadbb7800 x1476571022409660/t0(0) o35->dolphin-MDT0000-mdc-ffff8820121a4c00@172.25.32.25@tcp:23/10 lens 392/4328 e 1 to 1 dl 1408373243 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 07:47:23 tscc-1-8 kernel: Lustre: dolphin-MDT0000-mdc-ffff8820121a4c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 07:47:23 tscc-1-8 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8820121a4c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:47:27 tscc-2-50 kernel: Lustre: 18486:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408373153/real 1408373153]  req@ffff8806d4ee4000 x1476518982062724/t0(0) o35->dolphin-MDT0000-mdc-ffff881065a37400@172.25.32.25@tcp:23/10 lens 392/4328 e 1 to 1 dl 1408373247 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 07:47:27 tscc-2-50 kernel: Lustre: dolphin-MDT0000-mdc-ffff881065a37400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 07:47:27 tscc-2-50 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881065a37400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:47:41 tscc-2-48 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882066596000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:47:44 tscc-2-51 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881067285c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:47:48 tscc-1-8 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8820121a4c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:47:50 tscc-3-41 kernel: Lustre: 51524:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408373176/real 1408373176]  req@ffff880111a83c00 x1476519856339228/t0(0) o35->dolphin-MDT0000-mdc-ffff882026020800@172.25.32.25@tcp:23/10 lens 392/4328 e 1 to 1 dl 1408373270 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 07:47:50 tscc-3-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026020800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 07:47:50 tscc-3-41 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882026020800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:47:50 tscc-1-10 kernel: Lustre: 50532:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408373179/real 1408373179]  req@ffff880d6773b000 x1476547077057100/t0(0) o35->dolphin-MDT0000-mdc-ffff88200fa59400@172.25.32.25@tcp:23/10 lens 392/4328 e 1 to 1 dl 1408373270 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 07:47:50 tscc-1-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff88200fa59400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 07:47:50 tscc-1-10 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88200fa59400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:47:52 tscc-2-50 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881065a37400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:48:06 tscc-2-48 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882066596000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:48:09 tscc-2-51 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881067285c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:48:13 tscc-1-8 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8820121a4c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:48:15 tscc-3-41 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882026020800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:48:15 tscc-1-10 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88200fa59400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:48:17 tscc-2-50 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881065a37400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:48:25 tscc-1-11 kernel: Lustre: 10710:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408373213/real 1408373213]  req@ffff8819ec68a000 x1476519742253960/t0(0) o36->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.25@tcp:12/10 lens 536/440 e 1 to 1 dl 1408373305 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 07:48:25 tscc-1-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026d94400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 07:48:25 tscc-1-11 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882026d94400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:48:31 tscc-2-48 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882066596000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:48:34 tscc-2-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff881067285c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 07:48:38 tscc-1-8 kernel: Lustre: dolphin-MDT0000-mdc-ffff8820121a4c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 07:48:40 tscc-3-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026020800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 07:48:40 tscc-1-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff88200fa59400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 07:48:42 tscc-2-50 kernel: Lustre: dolphin-MDT0000-mdc-ffff881065a37400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 07:48:50 tscc-1-11 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882026d94400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:48:56 tscc-2-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff882066596000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 07:49:15 tscc-1-11 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882026d94400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:49:18 tscc-0-56 kernel: Lustre: 41751:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408373281/real 1408373281]  req@ffff880262a22800 x1476612995130988/t0(0) o101->dolphin-MDT0000-mdc-ffff8808335d7800@172.25.32.25@tcp:12/10 lens 600/2536 e 2 to 1 dl 1408373358 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 07:49:18 tscc-0-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808335d7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 07:49:18 tscc-0-56 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8808335d7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:49:18 tscc-0-56 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 07:49:40 tscc-1-11 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882026d94400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:49:43 tscc-0-56 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8808335d7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:50:05 tscc-1-11 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882026d94400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:50:08 tscc-0-56 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8808335d7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:50:30 tscc-1-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026d94400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 07:50:33 tscc-0-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808335d7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 07:51:17 tscc-3-41 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408373470/real 1408373470]  req@ffff881c8822e400 x1476519856346380/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408373477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 07:51:17 tscc-3-41 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 07:51:19 tscc-3-41 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 07:53:30 tscc-0-56 kernel: Lustre: 41751:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408373433/real 1408373433]  req@ffff880262a22800 x1476612995130988/t0(0) o101->dolphin-MDT0000-mdc-ffff8808335d7800@172.25.32.25@tcp:12/10 lens 600/2536 e 3 to 1 dl 1408373610 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Aug 18 07:53:30 tscc-0-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808335d7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 07:53:30 tscc-0-56 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8808335d7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:53:55 tscc-0-56 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8808335d7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 07:54:20 tscc-0-56 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8808335d7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 07:54:20 tscc-0-56 kernel: LustreError: 25508:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 07:54:20 tscc-0-56 kernel: LustreError: 51285:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x2000050a7:0xb2f9:0x0] error -108.
Aug 18 07:54:20 tscc-0-56 kernel: LustreError: 51385:0:(file.c:171:ll_close_inode_openhandle()) inode 402159236 mdc close failed: rc = -108
Aug 18 07:54:20 tscc-0-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808335d7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 08:09:22 tscc-0-59 kernel: LustreError: 28177:0:(dir.c:422:ll_get_dir_page()) read cache page: [0x20000515f:0xc68e:0x0] at 5596193933787687691: rc -5
Aug 18 08:16:07 tscc-0-59 kernel: LustreError: 30013:0:(dir.c:422:ll_get_dir_page()) read cache page: [0x20000515f:0xc990:0x0] at 5596193933787687691: rc -5
Aug 18 08:24:21 tscc-2-51 kernel: Lustre: 14272:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408375370/real 1408375370]  req@ffff881d38760c00 x1476518959633380/t0(0) o35->dolphin-MDT0000-mdc-ffff881067285c00@172.25.32.25@tcp:23/10 lens 392/4328 e 1 to 1 dl 1408375461 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 08:24:21 tscc-2-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff881067285c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 08:24:21 tscc-2-51 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881067285c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:24:44 tscc-0-59 kernel: Lustre: 32115:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408375393/real 1408375393]  req@ffff880b7064f000 x1476719731506744/t0(0) o36->dolphin-MDT0000-mdc-ffff880832bf5400@172.25.32.25@tcp:12/10 lens 520/4328 e 1 to 1 dl 1408375484 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 08:24:44 tscc-0-59 kernel: Lustre: dolphin-MDT0000-mdc-ffff880832bf5400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 08:24:44 tscc-0-59 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880832bf5400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:24:44 tscc-0-59 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 08:24:46 tscc-2-51 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881067285c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:25:09 tscc-0-59 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880832bf5400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:25:11 tscc-2-51 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881067285c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:25:34 tscc-0-59 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880832bf5400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:25:36 tscc-2-51 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881067285c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:25:39 tscc-3-54 kernel: Lustre: 2989:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408375419/real 1408375419]  req@ffff880c08e5c800 x1476290785030304/t0(0) o36->dolphin-MDT0000-mdc-ffff881026a4a400@172.25.32.25@tcp:12/10 lens 496/4328 e 2 to 1 dl 1408375539 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 08:25:39 tscc-3-54 kernel: Lustre: dolphin-MDT0000-mdc-ffff881026a4a400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 08:25:39 tscc-3-54 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881026a4a400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:25:59 tscc-0-59 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880832bf5400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:26:00 tscc-3-54 kernel: Lustre: 2658:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408375440/real 1408375440]  req@ffff881efdba1000 x1476290785034668/t0(0) o101->dolphin-MDT0000-mdc-ffff881026a4a400@172.25.32.25@tcp:12/10 lens 376/1928 e 1 to 1 dl 1408375560 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 08:26:01 tscc-2-51 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881067285c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:26:04 tscc-3-54 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881026a4a400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:26:24 tscc-0-59 kernel: Lustre: dolphin-MDT0000-mdc-ffff880832bf5400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 08:26:26 tscc-2-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff881067285c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 08:26:29 tscc-3-54 kernel: Lustre: dolphin-MDT0000-mdc-ffff881026a4a400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 08:34:21 tscc-0-59 kernel: LustreError: 35259:0:(dir.c:422:ll_get_dir_page()) read cache page: [0x20000515f:0xd268:0x0] at 5596193933787687691: rc -5
Aug 18 08:42:49 tscc-0-59 kernel: LustreError: 38764:0:(dir.c:422:ll_get_dir_page()) read cache page: [0x20000515f:0xd876:0x0] at 5596193933787687691: rc -5
Aug 18 08:43:19 tscc-1-53 kernel: LustreError: 11356:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -2
Aug 18 08:51:37 tscc-0-59 kernel: LustreError: 42596:0:(dir.c:422:ll_get_dir_page()) read cache page: [0x20000515f:0xdf55:0x0] at 5596193933787687691: rc -5
Aug 18 08:58:36 tscc-1-7 kernel: Lustre: 9114:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408377439/real 1408377439]  req@ffff88195db58c00 x1476519666295868/t0(0) o35->dolphin-MDT0000-mdc-ffff88101d811000@172.25.32.25@tcp:23/10 lens 392/4328 e 2 to 1 dl 1408377516 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 08:58:36 tscc-1-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101d811000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 08:58:36 tscc-1-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88101d811000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:58:36 tscc-0-56 kernel: Lustre: 54041:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408377439/real 1408377439]  req@ffff88028176b000 x1476612995537032/t0(0) o101->dolphin-MDT0000-mdc-ffff8808335d7800@172.25.32.25@tcp:12/10 lens 616/1136 e 2 to 1 dl 1408377516 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1
Aug 18 08:58:36 tscc-0-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808335d7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 08:58:36 tscc-0-56 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8808335d7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:58:37 tscc-1-7 kernel: Lustre: 9121:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408377440/real 1408377440]  req@ffff881027864800 x1476519666295888/t0(0) o101->dolphin-MDT0000-mdc-ffff88101d811000@172.25.32.25@tcp:12/10 lens 616/1136 e 2 to 1 dl 1408377517 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1
Aug 18 08:58:37 tscc-0-59 kernel: Lustre: 43378:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408377440/real 1408377440]  req@ffff880e59af2400 x1476719732241328/t0(0) o36->dolphin-MDT0000-mdc-ffff880832bf5400@172.25.32.25@tcp:12/10 lens 528/568 e 2 to 1 dl 1408377517 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 08:58:37 tscc-0-59 kernel: Lustre: dolphin-MDT0000-mdc-ffff880832bf5400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 08:58:37 tscc-0-59 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880832bf5400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:01 tscc-1-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88101d811000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:01 tscc-0-56 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8808335d7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:02 tscc-0-59 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880832bf5400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:09 tscc-1-36 kernel: Lustre: 63255:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408377458/real 1408377458]  req@ffff882027b85000 x1476569908913116/t0(0) o36->dolphin-MDT0000-mdc-ffff881028083800@172.25.32.25@tcp:12/10 lens 496/4328 e 1 to 1 dl 1408377549 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 08:59:09 tscc-1-36 kernel: Lustre: dolphin-MDT0000-mdc-ffff881028083800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 08:59:09 tscc-1-36 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881028083800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:10 tscc-2-51 kernel: Lustre: 30843:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408377473/real 1408377473]  req@ffff880bdc33c000 x1476518959750380/t0(0) o35->dolphin-MDT0000-mdc-ffff881067285c00@172.25.32.25@tcp:23/10 lens 392/4328 e 2 to 1 dl 1408377550 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 08:59:10 tscc-2-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff881067285c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 08:59:10 tscc-2-51 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881067285c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:11 tscc-1-10 kernel: Lustre: 60232:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408377460/real 1408377460]  req@ffff881fffc96000 x1476547077223348/t0(0) o36->dolphin-MDT0000-mdc-ffff88200fa59400@172.25.32.25@tcp:12/10 lens 496/4328 e 1 to 1 dl 1408377551 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 08:59:11 tscc-1-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff88200fa59400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 08:59:11 tscc-1-10 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88200fa59400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:26 tscc-1-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88101d811000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:26 tscc-0-56 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8808335d7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:27 tscc-0-59 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880832bf5400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:34 tscc-1-36 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881028083800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:35 tscc-2-51 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881067285c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:36 tscc-1-10 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88200fa59400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:49 tscc-1-13 kernel: Lustre: 4885:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408377498/real 1408377498]  req@ffff881c6dbe1400 x1476519650004852/t0(0) o35->dolphin-MDT0000-mdc-ffff8820293ac800@172.25.32.25@tcp:23/10 lens 392/4328 e 1 to 1 dl 1408377589 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 08:59:49 tscc-1-13 kernel: Lustre: dolphin-MDT0000-mdc-ffff8820293ac800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 08:59:49 tscc-1-13 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8820293ac800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:51 tscc-1-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88101d811000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:51 tscc-0-56 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8808335d7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:52 tscc-0-59 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880832bf5400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 08:59:59 tscc-1-36 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881028083800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 09:00:00 tscc-2-51 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881067285c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 09:00:01 tscc-1-10 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88200fa59400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 09:00:14 tscc-1-13 kernel: Lustre: dolphin-MDT0000-mdc-ffff8820293ac800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:00:16 tscc-1-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101d811000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:00:16 tscc-0-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808335d7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:00:17 tscc-0-59 kernel: Lustre: dolphin-MDT0000-mdc-ffff880832bf5400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:00:24 tscc-1-36 kernel: Lustre: dolphin-MDT0000-mdc-ffff881028083800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:00:25 tscc-2-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff881067285c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:00:26 tscc-1-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff88200fa59400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:03:43 tscc-0-59 kernel: LustreError: 47075:0:(dir.c:422:ll_get_dir_page()) read cache page: [0x20000515f:0xe711:0x0] at 5596193933787687691: rc -5
Aug 18 09:25:04 tscc-3-56 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379097/real 1408379097]  req@ffff8805d8d92000 x1476519829248444/t0(0) o400->dolphin-MDT0000-mdc-ffff8808317cc800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379104 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:04 tscc-3-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808317cc800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:04 tscc-3-56 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:04 tscc-1-32 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379097/real 1408379097]  req@ffff881db3d6b000 x1476439898179316/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379104 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:04 tscc-1-32 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:04 tscc-1-32 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810258fb000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:04 tscc-0-15 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379097/real 1408379097]  req@ffff88083083a000 x1476612876235356/t0(0) o400->dolphin-MDT0000-mdc-ffff8808300d3800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379104 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:04 tscc-0-15 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808300d3800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:05 tscc-0-31 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379098/real 1408379098]  req@ffff88025df7c000 x1476612857909876/t0(0) o400->dolphin-MDT0000-mdc-ffff880830a6b000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379105 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:05 tscc-0-31 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830a6b000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:05 tscc-0-31 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:05 tscc-0-28 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379098/real 1408379098]  req@ffff881005f81c00 x1476547206427836/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379105 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:05 tscc-0-28 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:05 tscc-0-28 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831447800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:05 tscc-1-6 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379098/real 1408379098]  req@ffff8804a847a000 x1476519511618988/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379105 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:05 tscc-1-6 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:05 tscc-1-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810266d7400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:05 tscc-3-51 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379098/real 1408379098]  req@ffff880c7b9bd800 x1476519881724452/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379105 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:05 tscc-3-51 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:05 tscc-3-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025aa6800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:05 tscc-0-36 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379098/real 1408379098]  req@ffff880826091400 x1476547204709572/t0(0) o400->dolphin-MDT0000-mdc-ffff880831f26c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379105 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:05 tscc-0-36 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831f26c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:05 tscc-0-36 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:10 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379104/real 1408379104]  req@ffff88053f64d000 x1476519829248708/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379110 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:10 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:10 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379104/real 1408379104]  req@ffff881deb05a400 x1476439898179580/t0(0) o38->dolphin-MDT0000-mdc-ffff8810258fb000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379110 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:10 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:10 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379104/real 1408379104]  req@ffff88083083a800 x1476612876235616/t0(0) o38->dolphin-MDT0000-mdc-ffff8808300d3800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379110 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:11 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379105/real 1408379105]  req@ffff880813451c00 x1476612857910140/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379111 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:11 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:11 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379105/real 1408379105]  req@ffff881009349000 x1476547206428104/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379111 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:11 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:11 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379105/real 1408379105]  req@ffff8820280c5800 x1476519511619256/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379111 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:11 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:11 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379105/real 1408379105]  req@ffff880d64dcb400 x1476519881724720/t0(0) o38->dolphin-MDT0000-mdc-ffff881025aa6800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379111 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:11 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:11 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379105/real 1408379105]  req@ffff8805e63a2800 x1476547204709836/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379111 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:11 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:12 tscc-1-11 kernel: Lustre: 2756:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379105/real 1408379105]  req@ffff881010a78400 x1476519742367436/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379112 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:12 tscc-1-11 kernel: Lustre: 2757:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379105/real 1408379105]  req@ffff8801d097e000 x1476519742367440/t0(0) o400->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379112 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:12 tscc-1-11 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:12 tscc-1-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026d94400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:12 tscc-0-57 kernel: Lustre: 2233:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379105/real 1408379105]  req@ffff881035981c00 x1475562021607168/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379112 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:12 tscc-0-57 kernel: Lustre: 2233:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:12 tscc-0-57 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:12 tscc-0-57 kernel: Lustre: dolphin-MDT0000-mdc-ffff881032828c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:14 tscc-1-1 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379107/real 1408379107]  req@ffff881a7de5f800 x1476519531589188/t0(0) o400->dolphin-MDT0000-mdc-ffff882026059c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379114 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:14 tscc-1-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026059c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:14 tscc-1-1 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:14 tscc-2-39 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379107/real 1408379107]  req@ffff88103126d800 x1476547206072996/t0(0) o400->dolphin-MDT0000-mdc-ffff8808290f1c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379114 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:14 tscc-2-39 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808290f1c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:14 tscc-2-39 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:15 tscc-3-40 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379108/real 1408379108]  req@ffff880e76998000 x1476520324301796/t0(0) o400->dolphin-MDT0000-mdc-ffff881025a8d400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:15 tscc-3-40 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 09:25:15 tscc-3-40 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025a8d400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:15 tscc-3-40 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:15 tscc-1-14 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379108/real 1408379108]  req@ffff88400ff99800 x1476519630497976/t0(0) o400->dolphin-MDT0000-mdc-ffff884010213000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:15 tscc-1-14 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 09:25:15 tscc-1-14 kernel: Lustre: dolphin-MDT0000-mdc-ffff884010213000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:15 tscc-1-14 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:15 tscc-0-20 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379108/real 1408379108]  req@ffff880819564c00 x1476638723775512/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:15 tscc-0-20 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:15 tscc-0-20 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810332bc400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:15 tscc-gpu-7-7 kernel: Lustre: 2541:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379108/real 1408379108]  req@ffff880877b08400 x1476536279980540/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:15 tscc-gpu-7-7 kernel: Lustre: 2540:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379108/real 1408379108]  req@ffff880877b08800 x1476536279980544/t0(0) o400->dolphin-MDT0000-mdc-ffff880876dd2c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:15 tscc-gpu-7-7 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:15 tscc-gpu-7-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876dd2c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:15 tscc-0-2 kernel: Lustre: 2756:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379108/real 1408379108]  req@ffff880ffd8fe400 x1476547435018612/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:15 tscc-0-2 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:15 tscc-1-61 kernel: Lustre: 2743:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379108/real 1408379108]  req@ffff8807190c6400 x1476536929472956/t0(0) o400->dolphin-MDT0000-mdc-ffff88103115e400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:15 tscc-1-61 kernel: Lustre: 2743:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:15 tscc-1-61 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:15 tscc-1-61 kernel: Lustre: dolphin-MDT0000-mdc-ffff88103115e400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:15 tscc-2-13 kernel: Lustre: 2968:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379108/real 1408379108]  req@ffff881c5c6d0000 x1476519055559948/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:15 tscc-2-13 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:15 tscc-2-13 kernel: Lustre: dolphin-MDT0000-mdc-ffff881065172800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:17 tscc-gpu-5-4 kernel: Lustre: 2631:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379110/real 1408379110]  req@ffff880876f0cc00 x1476612337674828/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379117 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:17 tscc-gpu-5-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:18 tscc-login1 kernel: Lustre: 2823:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379111/real 1408379111]  req@ffff8802ce02b000 x1476710234789632/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379118 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:18 tscc-login1 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:18 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379112/real 1408379112]  req@ffff880a1abde000 x1476519742367704/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379118 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:18 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379112/real 1408379112]  req@ffff881011c77400 x1475562021607436/t0(0) o38->dolphin-MDT0000-mdc-ffff881032828c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379118 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:18 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:20 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379114/real 1408379114]  req@ffff881a57ffc400 x1476519531589452/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379120 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:20 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:20 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379114/real 1408379114]  req@ffff88080e2f7c00 x1476547206073260/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379120 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:20 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:21 tscc-1-48 kernel: Lustre: 2735:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379114/real 1408379114]  req@ffff880cfd20bc00 x1476537001787028/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379121 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:21 tscc-1-48 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:21 tscc-1-45 kernel: Lustre: 2749:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379111/real 1408379111]  req@ffff88102875e800 x1476537015431704/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379121 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:21 tscc-1-45 kernel: Lustre: 2749:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 09:25:21 tscc-1-45 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:21 tscc-0-10 kernel: Lustre: 2759:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379113/real 1408379113]  req@ffff880824457c00 x1476547227388716/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379121 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:21 tscc-0-10 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:21 tscc-1-40 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379111/real 1408379111]  req@ffff880ee5917400 x1476537020691312/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379121 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:21 tscc-1-40 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:21 tscc-3-43 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379111/real 1408379111]  req@ffff8806ba630000 x1476519948816940/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379121 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:21 tscc-3-43 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 09:25:21 tscc-3-43 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:21 tscc-3-48 kernel: Lustre: 2750:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379114/real 1408379114]  req@ffff881bec497000 x1476287317195172/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379121 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:21 tscc-3-48 kernel: Lustre: 2750:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:21 tscc-0-20 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379115/real 1408379115]  req@ffff880816571400 x1476638723775776/t0(0) o38->dolphin-MDT0000-mdc-ffff8810332bc400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379121 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:21 tscc-0-20 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:21 tscc-3-48 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:21 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379115/real 1408379115]  req@ffff880876146400 x1476536279980804/t0(0) o38->dolphin-MDT0000-mdc-ffff880876dd2c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379121 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:21 tscc-0-2 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379115/real 1408379115]  req@ffff880ffd1cf400 x1476547435018876/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379121 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:21 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379115/real 1408379115]  req@ffff8808336f4c00 x1476536929473220/t0(0) o38->dolphin-MDT0000-mdc-ffff88103115e400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379121 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:21 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379115/real 1408379115]  req@ffff881dc3bfb400 x1476519055560572/t0(0) o38->dolphin-MDT0000-mdc-ffff881065172800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379121 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:21 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:23 tscc-gpu-5-4 kernel: Lustre: 2620:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379117/real 1408379117]  req@ffff8808073b5400 x1476612337675092/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379123 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:23 tscc-1-56 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379116/real 1408379116]  req@ffff882026941400 x1476536956336372/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379123 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:23 tscc-1-56 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:24 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379118/real 1408379118]  req@ffff88012c39a800 x1476710234789884/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379124 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:25 tscc-0-7 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379118/real 1408379118]  req@ffff881024e49000 x1476547271337380/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379125 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:25 tscc-0-7 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:25 tscc-1-35 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379118/real 1408379118]  req@ffff880e21043000 x1476285713736728/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379125 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:25 tscc-1-35 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:25 tscc-1-53 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379118/real 1408379118]  req@ffff880f69f8c000 x1476537036420500/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379125 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:25 tscc-1-53 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:25 tscc-0-62 kernel: Lustre: 2473:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379118/real 1408379118]  req@ffff8808315a5000 x1475809553493268/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379125 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:25 tscc-0-62 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:25 tscc-2-52 kernel: Lustre: 2970:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379118/real 1408379118]  req@ffff8802df87dc00 x1476290266794732/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379125 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:25 tscc-2-52 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:25 tscc-2-10 kernel: Lustre: 2974:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379118/real 1408379118]  req@ffff88099bb01c00 x1476519079645084/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379125 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:25 tscc-2-10 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:27 tscc-1-48 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379121/real 1408379121]  req@ffff880eb0bac400 x1476537001787292/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379127 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:28 tscc-0-10 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379121/real 1408379121]  req@ffff880826dc4000 x1476547227388980/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379128 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:29 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379123/real 1408379123]  req@ffff8820261e4400 x1476536956336636/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379129 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:30 tscc-1-45 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379121/real 1408379121]  req@ffff880ed9c51c00 x1476537015431968/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379130 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:30 tscc-1-40 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379121/real 1408379121]  req@ffff880eb6f80800 x1476537020691576/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379130 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:30 tscc-3-43 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379121/real 1408379121]  req@ffff880ca4ed4400 x1476519948817204/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379130 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:31 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379125/real 1408379125]  req@ffff881030aaf400 x1476547271337644/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379131 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:31 tscc-1-35 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379125/real 1408379125]  req@ffff880e999e8000 x1476285713736992/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379131 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:31 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379125/real 1408379125]  req@ffff880d2156a400 x1476537036420764/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379131 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:31 tscc-0-62 kernel: Lustre: 2458:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379125/real 1408379125]  req@ffff880831b0b800 x1475809553493532/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379131 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:31 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379125/real 1408379125]  req@ffff880cab493800 x1476290266795344/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379131 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:31 tscc-2-10 kernel: Lustre: 2962:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379125/real 1408379125]  req@ffff8808eb086000 x1476519079645716/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379131 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:35 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379129/real 1408379129]  req@ffff88081452ac00 x1476519829248712/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379135 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:35 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:35 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379129/real 1408379129]  req@ffff882022821000 x1476439898179592/t0(0) o38->dolphin-MDT0000-mdc-ffff8810258fb000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379135 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:35 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:35 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379129/real 1408379129]  req@ffff88083083a000 x1476612876235624/t0(0) o38->dolphin-MDT0000-mdc-ffff8808300d3800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379135 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:36 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379130/real 1408379130]  req@ffff8808128c5800 x1476612857910148/t0(0) o38->dolphin-MDT0000-mdc-ffff880830a6b000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379136 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:36 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:36 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379130/real 1408379130]  req@ffff88102fd2fc00 x1476547206428112/t0(0) o38->dolphin-MDT0000-mdc-ffff880831447800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379136 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:36 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:36 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379130/real 1408379130]  req@ffff881d45bb3400 x1476519511619264/t0(0) o38->dolphin-MDT0000-mdc-ffff8810266d7400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379136 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:36 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:36 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379130/real 1408379130]  req@ffff880d64dcb800 x1476519881724728/t0(0) o38->dolphin-MDT0000-mdc-ffff881025aa6800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379136 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:36 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:36 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379130/real 1408379130]  req@ffff880826091000 x1476547204709844/t0(0) o38->dolphin-MDT0000-mdc-ffff880831f26c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379136 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:36 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:41 tscc-0-15 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379129/real 1408379129]  req@ffff88083083a400 x1476612876235620/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408379141 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:41 tscc-0-15 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 09:25:43 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379137/real 1408379137]  req@ffff880c29913000 x1476519742367712/t0(0) o38->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379143 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:43 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:43 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379137/real 1408379137]  req@ffff881008685800 x1475562021607444/t0(0) o38->dolphin-MDT0000-mdc-ffff881032828c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379143 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:43 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:44 tscc-login1 kernel: Lustre: 34435:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379100/real 1408379100]  req@ffff880ca3e98c00 x1476710234789628/t0(0) o101->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.25@tcp:12/10 lens 584/2536 e 0 to 1 dl 1408379144 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 09:25:44 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:45 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379139/real 1408379139]  req@ffff881afba04800 x1476519531589460/t0(0) o38->dolphin-MDT0000-mdc-ffff882026059c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379145 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:45 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:45 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379139/real 1408379139]  req@ffff8808245a0000 x1476547206073268/t0(0) o38->dolphin-MDT0000-mdc-ffff8808290f1c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379145 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:45 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:46 tscc-3-40 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379140/real 1408379140]  req@ffff880f98737c00 x1476520324302068/t0(0) o38->dolphin-MDT0000-mdc-ffff881025a8d400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379146 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:46 tscc-3-40 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 09:25:46 tscc-1-14 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379140/real 1408379140]  req@ffff88400fe5e400 x1476519630498248/t0(0) o38->dolphin-MDT0000-mdc-ffff884010213000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379146 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:46 tscc-1-14 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 09:25:46 tscc-0-20 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379140/real 1408379140]  req@ffff88081a9d7c00 x1476638723775784/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379146 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:46 tscc-0-20 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:46 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379140/real 1408379140]  req@ffff880875b1f400 x1476536279980816/t0(0) o38->dolphin-MDT0000-mdc-ffff880876dd2c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379146 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:46 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:46 tscc-0-2 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379140/real 1408379140]  req@ffff880fe7ab6000 x1476547435018880/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379146 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:46 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379140/real 1408379140]  req@ffff8807190c6800 x1476536929473224/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379146 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:46 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:46 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379140/real 1408379140]  req@ffff881e54a14000 x1476519055561876/t0(0) o38->dolphin-MDT0000-mdc-ffff881065172800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379146 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:46 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:48 tscc-gpu-5-4 kernel: Lustre: 2620:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379142/real 1408379142]  req@ffff88086fd5b800 x1476612337675096/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379148 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:49 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379143/real 1408379143]  req@ffff8802ce02b000 x1476710234789896/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379149 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:52 tscc-1-48 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379146/real 1408379146]  req@ffff880cfd20bc00 x1476537001787296/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379152 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:52 tscc-3-48 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379146/real 1408379146]  req@ffff881a0dea8400 x1476287317195440/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379152 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:52 tscc-3-48 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:25:52 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379141/real 1408379141]  req@ffff88021e505c00 x1476612876235884/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379152 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:53 tscc-0-10 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379146/real 1408379146]  req@ffff880824457000 x1476547227388984/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379153 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:54 tscc-gpu-5-4 kernel: Lustre: 2621:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379110/real 1408379110]  req@ffff880878099400 x1476612337674832/t0(0) o400->dolphin-MDT0000-mdc-ffff880877501800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379154 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:54 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:54 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379148/real 1408379148]  req@ffff8819d677c400 x1476536956336640/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379154 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:55 tscc-1-45 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379146/real 1408379146]  req@ffff88102875e400 x1476537015431972/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379155 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:55 tscc-1-40 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379146/real 1408379146]  req@ffff880f87d55c00 x1476537020691580/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379155 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:55 tscc-3-43 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379146/real 1408379146]  req@ffff880ff49b3400 x1476519948817208/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379155 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:56 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379150/real 1408379150]  req@ffff88081ca91c00 x1476547271337648/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379156 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:56 tscc-1-35 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379150/real 1408379150]  req@ffff880e999e8800 x1476285713736996/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379156 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:56 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379150/real 1408379150]  req@ffff880f69f8cc00 x1476537036420860/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379156 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:56 tscc-0-62 kernel: Lustre: 2458:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379150/real 1408379150]  req@ffff88076c6afc00 x1475809553493536/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379156 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:56 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379150/real 1408379150]  req@ffff88067d15ec00 x1476290266796552/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379156 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:56 tscc-2-10 kernel: Lustre: 2962:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379150/real 1408379150]  req@ffff880bd5ff9c00 x1476519079646984/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379156 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:56 tscc-0-2 kernel: Lustre: 2758:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379108/real 1408379108]  req@ffff880ffbb2b400 x1476547435018616/t0(0) o400->dolphin-MDT0000-mdc-ffff88083175c400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379156 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:56 tscc-0-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff88083175c400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:58 tscc-1-48 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379114/real 1408379114]  req@ffff880cfd20b800 x1476537001787032/t0(0) o400->dolphin-MDT0000-mdc-ffff882028f17800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379158 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:58 tscc-1-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028f17800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:58 tscc-1-45 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028c86c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:58 tscc-0-10 kernel: Lustre: 2760:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379113/real 1408379113]  req@ffff880827638800 x1476547227388720/t0(0) o400->dolphin-MDT0000-mdc-ffff8810322ef000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379158 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:25:58 tscc-0-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810322ef000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:58 tscc-3-43 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101eab6000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:25:58 tscc-3-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810266f7400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:26:00 tscc-1-56 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379116/real 1408379116]  req@ffff882026941000 x1476536956336376/t0(0) o400->dolphin-MDT0000-mdc-ffff882028ff0000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379160 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:26:00 tscc-1-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028ff0000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:26:02 tscc-0-7 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379118/real 1408379118]  req@ffff881024e49400 x1476547271337384/t0(0) o400->dolphin-MDT0000-mdc-ffff880833252000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379162 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:26:02 tscc-0-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880833252000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:26:02 tscc-1-35 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379118/real 1408379118]  req@ffff881025add400 x1476285713736732/t0(0) o400->dolphin-MDT0000-mdc-ffff881024c5a000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379162 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:26:02 tscc-1-35 kernel: Lustre: dolphin-MDT0000-mdc-ffff881024c5a000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:26:02 tscc-1-53 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379118/real 1408379118]  req@ffff880f69f8c800 x1476537036420504/t0(0) o400->dolphin-MDT0000-mdc-ffff882026316400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379162 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:26:02 tscc-1-53 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026316400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:26:02 tscc-0-62 kernel: Lustre: 2474:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379118/real 1408379118]  req@ffff880832fd2800 x1475809553493272/t0(0) o400->dolphin-MDT0000-mdc-ffff881033ef6400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379162 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:26:02 tscc-0-62 kernel: Lustre: dolphin-MDT0000-mdc-ffff881033ef6400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:26:02 tscc-2-10 kernel: Lustre: 2974:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379118/real 1408379118]  req@ffff88099bb01800 x1476519079645088/t0(0) o400->dolphin-MDT0000-mdc-ffff881066654400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379162 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:26:02 tscc-2-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff881066654400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:26:02 tscc-2-52 kernel: Lustre: 2971:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379118/real 1408379118]  req@ffff8806b00df400 x1476290266794736/t0(0) o400->dolphin-MDT0000-mdc-ffff881067a76000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379162 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:26:02 tscc-2-52 kernel: Lustre: dolphin-MDT0000-mdc-ffff881067a76000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:26:04 tscc-1-48 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379158/real 1408379158]  req@ffff880cfd20bc00 x1476537001787560/t0(0) o38->dolphin-MDT0000-mdc-ffff882028f17800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379164 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:26:04 tscc-1-40 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379111/real 1408379111]  req@ffff880f62dde000 x1476537020691316/t0(0) o400->dolphin-MDT0000-mdc-ffff881025cb7400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408379164 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:26:04 tscc-1-40 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025cb7400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:26:04 tscc-1-32 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-3-56 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-3-43 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101eab6000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-3-43 kernel: Lustre: Skipped 1 previous similar message
Aug 18 09:26:04 tscc-2-39 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-0-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff88083175c400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-1-1 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-1-1 kernel: Lustre: Skipped 1 previous similar message
Aug 18 09:26:04 tscc-1-40 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025cb7400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-2-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff881066654400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-0-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880833252000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-0-57 kernel: Lustre: dolphin-MDT0000-mdc-ffff881032828c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-0-57 kernel: Lustre: Skipped 1 previous similar message
Aug 18 09:26:04 tscc-3-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810266f7400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-2-52 kernel: Lustre: dolphin-MDT0000-mdc-ffff881067a76000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-1-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028ff0000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-1-53 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026316400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-1-35 kernel: Lustre: dolphin-MDT0000-mdc-ffff881024c5a000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-0-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810322ef000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-1-45 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028c86c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-1-45 kernel: Lustre: Skipped 1 previous similar message
Aug 18 09:26:04 tscc-0-62 kernel: Lustre: dolphin-MDT0000-mdc-ffff881033ef6400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-1-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026d94400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-3-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025aa6800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-0-28 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831447800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-0-31 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830a6b000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-0-36 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831f26c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-1-6 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:04 tscc-1-6 kernel: Lustre: Skipped 1 previous similar message
Aug 18 09:26:04 tscc-0-57 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:05 tscc-3-40 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025a8d400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:05 tscc-3-40 kernel: Lustre: Skipped 1 previous similar message
Aug 18 09:26:05 tscc-1-14 kernel: Lustre: dolphin-MDT0000-mdc-ffff884010213000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:05 tscc-1-14 kernel: Lustre: Skipped 1 previous similar message
Aug 18 09:26:05 tscc-0-20 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810332bc400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:05 tscc-gpu-7-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876dd2c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:05 tscc-1-61 kernel: Lustre: dolphin-MDT0000-mdc-ffff88103115e400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:05 tscc-2-13 kernel: Lustre: dolphin-MDT0000-mdc-ffff881065172800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:06 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408379166/real 1408379166]  req@ffff88083083a800 x1476612876235888/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408379177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:26:06 tscc-0-15 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808300d3800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:09 tscc-login1 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:19 tscc-gpu-5-4 kernel: Lustre: 2620:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408379179/real 1408379179]  req@ffff88086fd5b800 x1476612337675368/t0(0) o38->dolphin-MDT0000-mdc-ffff880877501800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379206 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:26:19 tscc-gpu-5-4 kernel: Lustre: 2620:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:26:19 tscc-gpu-5-4 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:21 tscc-0-2 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:23 tscc-1-48 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408379183/real 1408379183]  req@ffff880cfd20bc00 x1476537001787568/t0(0) o38->dolphin-MDT0000-mdc-ffff882028f17800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379206 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:26:23 tscc-1-48 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:23 tscc-1-45 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:23 tscc-0-10 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:23 tscc-3-43 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:23 tscc-3-48 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:25 tscc-1-56 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:27 tscc-0-7 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:27 tscc-1-35 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:27 tscc-1-53 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:27 tscc-0-62 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:27 tscc-2-10 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:27 tscc-2-52 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:29 tscc-1-40 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:31 tscc-0-15 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 09:26:34 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408379194/real 1408379194]  req@ffff8802d5a79c00 x1476710234790460/t0(0) o38->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408379220 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 09:26:34 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 09:26:44 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:48 tscc-1-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028f17800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:26:59 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:36:41 tscc-0-59 kernel: Lustre: 63608:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408379724/real 1408379724]  req@ffff8805e96f0c00 x1476719733331592/t0(0) o36->dolphin-MDT0000-mdc-ffff880832bf5400@172.25.32.25@tcp:12/10 lens 520/4328 e 2 to 1 dl 1408379801 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 09:36:41 tscc-0-59 kernel: Lustre: dolphin-MDT0000-mdc-ffff880832bf5400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:36:41 tscc-0-59 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880832bf5400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 09:37:06 tscc-0-59 kernel: Lustre: dolphin-MDT0000-mdc-ffff880832bf5400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 09:37:24 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation obd_ping failed with -107.
Aug 18 09:37:24 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 09:37:24 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 09:37:24 tscc-login1 kernel: LustreError: 37761:0:(file.c:171:ll_close_inode_openhandle()) inode 144115524375203316 mdc close failed: rc = -108
Aug 18 09:37:24 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:03:11 tscc-0-59 kernel: LustreError: 14297:0:(dir.c:422:ll_get_dir_page()) read cache page: [0x20000515f:0x1215d:0x0] at 5596193933787687691: rc -5
Aug 18 10:11:45 tscc-0-63 kernel: Lustre: Unmounted dolphin-client
Aug 18 10:17:44 tscc-0-59 kernel: LustreError: 23281:0:(dir.c:422:ll_get_dir_page()) read cache page: [0x20000515f:0x131ca:0x0] at 5596193933787687691: rc -5
Aug 18 10:23:33 tscc-0-23 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408382606/real 1408382606]  req@ffff88100467f400 x1476547159861128/t0(0) o400->dolphin-MDT0000-mdc-ffff881030c9b000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408382613 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:23:33 tscc-0-23 kernel: Lustre: dolphin-MDT0000-mdc-ffff881030c9b000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:23:39 tscc-0-23 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408382613/real 1408382613]  req@ffff880ff3abac00 x1476547159861388/t0(0) o38->dolphin-MDT0000-mdc-ffff881030c9b000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408382619 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:24:04 tscc-0-23 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408382638/real 1408382638]  req@ffff88101fb4bc00 x1476547159861396/t0(0) o38->dolphin-MDT0000-mdc-ffff881030c9b000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408382644 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:24:23 tscc-0-23 kernel: Lustre: dolphin-MDT0000-mdc-ffff881030c9b000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:32:58 tscc-gpu-3-3 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 10:33:03 tscc-gpu-3-3 kernel: Lustre: 2942:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383178/real 1408383178]  req@ffff881070d6ec00 x1476796799254532/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383183 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:33:04 tscc-gpu-3-3 kernel: LustreError: 2878:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff881070d6e800 x1476796799254536/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 10:33:11 tscc-gpu-3-3 kernel: LustreError: 2972:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff881070d6e000 x1476796799254544/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 10:33:28 tscc-gpu-3-3 kernel: Lustre: 2942:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383203/real 1408383203]  req@ffff8810713a7c00 x1476796799254628/t0(0) o38->dolphin-MDT0000-mdc-ffff8810711ad800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383208 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:33:48 tscc-gpu-3-3 kernel: Lustre: Layout lock feature supported.
Aug 18 10:33:48 tscc-gpu-3-3 kernel: Lustre: Mounted dolphin-client
Aug 18 10:35:25 tscc-gpu-3-3 kernel: Lustre: Unmounted dolphin-client
Aug 18 10:39:36 tscc-login1 kernel: Lustre: 2827:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383569/real 1408383569]  req@ffff880cb130cc00 x1476710234919860/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383576 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:39:36 tscc-login1 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:39:37 tscc-login1 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 10:42:59 tscc-login1 kernel: Lustre: 19271:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383769/real 1408383769]  req@ffff88001240a000 x1476710234922920/t0(0) o35->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408383779 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 10:42:59 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:07 tscc-0-15 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383768/real 1408383768]  req@ffff880acf199400 x1476612876380892/t0(0) o103->dolphin-MDT0000-mdc-ffff8808300d3800@172.25.32.25@tcp:17/18 lens 328/224 e 0 to 1 dl 1408383787 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 10:43:07 tscc-0-15 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808300d3800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:08 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383779/real 1408383779]  req@ffff880e87fd0800 x1476710234923232/t0(0) o38->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383788 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:08 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:25 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383787/real 1408383787]  req@ffff88054f472400 x1476612876380900/t0(0) o38->dolphin-MDT0000-mdc-ffff8808300d3800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383805 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:25 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:29 tscc-1-40 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383764/real 1408383764]  req@ffff881f020e9000 x1476537020750744/t0(0) o400->dolphin-MDT0000-mdc-ffff881025cb7400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:29 tscc-1-40 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025cb7400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:29 tscc-3-40 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383765/real 1408383765]  req@ffff881009581000 x1476520324363244/t0(0) o400->dolphin-MDT0000-mdc-ffff881025a8d400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:29 tscc-3-40 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:29 tscc-3-40 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025a8d400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:29 tscc-1-14 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383765/real 1408383765]  req@ffff883fed0eb800 x1476519630547120/t0(0) o400->dolphin-MDT0000-mdc-ffff884010213000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:29 tscc-1-14 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:29 tscc-1-14 kernel: Lustre: dolphin-MDT0000-mdc-ffff884010213000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:29 tscc-0-20 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383765/real 1408383765]  req@ffff881034393000 x1476638723824660/t0(0) o400->dolphin-MDT0000-mdc-ffff8810332bc400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:29 tscc-0-20 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:29 tscc-0-20 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810332bc400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:29 tscc-gpu-7-7 kernel: Lustre: 2547:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383765/real 1408383765]  req@ffff880458cef800 x1476536280029688/t0(0) o400->dolphin-MDT0000-mdc-ffff880876dd2c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:29 tscc-gpu-7-7 kernel: Lustre: 2547:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:29 tscc-gpu-7-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876dd2c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:29 tscc-1-61 kernel: Lustre: 2740:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383765/real 1408383765]  req@ffff8807269e6800 x1476536929544516/t0(0) o400->dolphin-MDT0000-mdc-ffff88103115e400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:29 tscc-1-61 kernel: Lustre: 2740:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:29 tscc-1-61 kernel: Lustre: dolphin-MDT0000-mdc-ffff88103115e400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:29 tscc-2-13 kernel: Lustre: 2979:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383765/real 1408383765]  req@ffff882067fb2000 x1476519055814800/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:29 tscc-2-13 kernel: Lustre: 2979:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:29 tscc-2-13 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:30 tscc-0-15 kernel: Lustre: 2735:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383766/real 1408383766]  req@ffff8806aa2b2400 x1476612876380864/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383810 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:30 tscc-0-15 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:33 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383804/real 1408383804]  req@ffff880b8d4df000 x1476710234923252/t0(0) o38->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383813 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:35 tscc-3-40 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383809/real 1408383809]  req@ffff8810248b8400 x1476520324363768/t0(0) o38->dolphin-MDT0000-mdc-ffff881025a8d400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:35 tscc-1-14 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383809/real 1408383809]  req@ffff88401272d000 x1476519630547644/t0(0) o38->dolphin-MDT0000-mdc-ffff884010213000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:35 tscc-0-20 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383809/real 1408383809]  req@ffff881034393800 x1476638723825184/t0(0) o38->dolphin-MDT0000-mdc-ffff8810332bc400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:35 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383809/real 1408383809]  req@ffff880459b16000 x1476536280030212/t0(0) o38->dolphin-MDT0000-mdc-ffff880876dd2c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:35 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383809/real 1408383809]  req@ffff88082967ec00 x1476536929545040/t0(0) o38->dolphin-MDT0000-mdc-ffff88103115e400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:35 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383809/real 1408383809]  req@ffff882067fb2400 x1476519055815328/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:36 tscc-1-40 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383809/real 1408383809]  req@ffff882005f91800 x1476537020751268/t0(0) o38->dolphin-MDT0000-mdc-ffff881025cb7400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383816 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:40 tscc-login1 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:43 tscc-3-56 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383779/real 1408383779]  req@ffff88070b5c3800 x1476519829324268/t0(0) o400->dolphin-MDT0000-mdc-ffff8808317cc800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383823 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:43 tscc-3-56 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:43 tscc-3-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808317cc800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:43 tscc-3-56 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:43 tscc-1-32 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383779/real 1408383779]  req@ffff881da1d3ec00 x1476439898236632/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383823 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:43 tscc-1-32 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:43 tscc-1-32 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:43 tscc-1-32 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810258fb000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:43 tscc-gpu-5-4 kernel: Lustre: 2626:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383779/real 1408383779]  req@ffff88046356b800 x1476612337738376/t0(0) o400->dolphin-MDT0000-mdc-ffff880877501800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383823 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:43 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:43 tscc-gpu-5-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:44 tscc-0-31 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383780/real 1408383780]  req@ffff880812ae2800 x1476612857959284/t0(0) o400->dolphin-MDT0000-mdc-ffff880830a6b000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383824 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:44 tscc-0-31 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:44 tscc-0-31 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830a6b000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:44 tscc-0-31 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:44 tscc-0-28 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383780/real 1408383780]  req@ffff88102fd2f800 x1476547206477248/t0(0) o400->dolphin-MDT0000-mdc-ffff880831447800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383824 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:44 tscc-0-28 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:44 tscc-0-28 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831447800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:44 tscc-0-28 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:44 tscc-0-15 kernel: Lustre: 18143:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383768/real 1408383768]  req@ffff880bf920b800 x1476612876380724/t0(0) o101->dolphin-MDT0000-mdc-ffff8808300d3800@172.25.32.25@tcp:12/10 lens 632/1136 e 0 to 1 dl 1408383824 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1
Aug 18 10:43:44 tscc-0-15 kernel: Lustre: 18143:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:47 tscc-1-48 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383783/real 1408383783]  req@ffff88101975c400 x1476537001864020/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:47 tscc-1-48 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:47 tscc-1-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028f17800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:47 tscc-1-45 kernel: Lustre: 2763:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383783/real 1408383783]  req@ffff880ee6212000 x1476537015491436/t0(0) o400->dolphin-MDT0000-mdc-ffff882028c86c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:47 tscc-1-45 kernel: Lustre: 2763:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:47 tscc-1-45 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028c86c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:47 tscc-0-10 kernel: Lustre: 2756:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383783/real 1408383783]  req@ffff880824457000 x1476547227437864/t0(0) o400->dolphin-MDT0000-mdc-ffff8810322ef000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:47 tscc-0-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810322ef000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:47 tscc-0-10 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:47 tscc-1-6 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383780/real 1408383780]  req@ffff8820263f1800 x1476519511691060/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:47 tscc-1-6 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:47 tscc-1-6 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:47 tscc-1-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810266d7400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:47 tscc-3-43 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383783/real 1408383783]  req@ffff880a1aed0c00 x1476519948866088/t0(0) o400->dolphin-MDT0000-mdc-ffff88101eab6000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:47 tscc-3-43 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:47 tscc-3-43 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101eab6000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:47 tscc-3-43 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:47 tscc-0-36 kernel: Lustre: 2739:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383780/real 1408383780]  req@ffff880827a82c00 x1476547204758980/t0(0) o400->dolphin-MDT0000-mdc-ffff880831f26c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:47 tscc-0-36 kernel: Lustre: 2739:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:47 tscc-0-36 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831f26c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:47 tscc-0-36 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:47 tscc-3-48 kernel: Lustre: 2754:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383783/real 1408383783]  req@ffff881bec497000 x1476287317244316/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:47 tscc-3-48 kernel: Lustre: 2754:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:47 tscc-3-48 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:47 tscc-3-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810266f7400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:47 tscc-3-51 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383780/real 1408383780]  req@ffff88100a9dcc00 x1476519881773864/t0(0) o400->dolphin-MDT0000-mdc-ffff881025aa6800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:47 tscc-3-51 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:47 tscc-3-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025aa6800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:47 tscc-3-51 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:47 tscc-0-2 kernel: Lustre: 2759:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383781/real 1408383781]  req@ffff880832d8b000 x1476547435067760/t0(0) o400->dolphin-MDT0000-mdc-ffff88083175c400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:47 tscc-0-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff88083175c400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:47 tscc-0-2 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:49 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383823/real 1408383823]  req@ffff880467becc00 x1476519829324792/t0(0) o38->dolphin-MDT0000-mdc-ffff8808317cc800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383829 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:49 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:49 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383823/real 1408383823]  req@ffff881d0513b000 x1476439898237160/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383829 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:49 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:49 tscc-gpu-5-4 kernel: Lustre: 2620:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383823/real 1408383823]  req@ffff88045d68d800 x1476612337738900/t0(0) o38->dolphin-MDT0000-mdc-ffff880877501800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383829 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:49 tscc-gpu-5-4 kernel: Lustre: 2620:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:49 tscc-1-56 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383785/real 1408383785]  req@ffff882013b98c00 x1476536956385516/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383829 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:49 tscc-1-56 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:49 tscc-1-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028ff0000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:50 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383824/real 1408383824]  req@ffff880817142c00 x1476612857959812/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383830 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:50 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:50 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383824/real 1408383824]  req@ffff881005fdd400 x1476547206477776/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383830 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:50 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:51 tscc-0-7 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383787/real 1408383787]  req@ffff880819fba800 x1476547271386528/t0(0) o400->dolphin-MDT0000-mdc-ffff880833252000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383831 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:51 tscc-0-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880833252000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:51 tscc-0-7 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:51 tscc-1-35 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383787/real 1408383787]  req@ffff8819e9fc2000 x1476285713801348/t0(0) o400->dolphin-MDT0000-mdc-ffff881024c5a000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383831 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:51 tscc-1-35 kernel: Lustre: dolphin-MDT0000-mdc-ffff881024c5a000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:51 tscc-1-35 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:51 tscc-1-53 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383787/real 1408383787]  req@ffff880fb514dc00 x1476537036477668/t0(0) o400->dolphin-MDT0000-mdc-ffff882026316400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383831 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:51 tscc-1-53 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026316400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:51 tscc-1-53 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:52 tscc-1-45 kernel: Lustre: 2763:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383783/real 1408383783]  req@ffff880d14a34400 x1476537015491432/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383832 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:52 tscc-1-45 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:53 tscc-1-48 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383827/real 1408383827]  req@ffff8810273b8c00 x1476537001864552/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383833 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:53 tscc-1-48 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:53 tscc-0-10 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383827/real 1408383827]  req@ffff880826d72800 x1476547227438392/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383833 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:53 tscc-0-10 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:53 tscc-3-43 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383827/real 1408383827]  req@ffff880a1aed0000 x1476519948866616/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383833 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:53 tscc-3-43 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:53 tscc-3-48 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383827/real 1408383827]  req@ffff881545813000 x1476287317244848/t0(0) o38->dolphin-MDT0000-mdc-ffff8810266f7400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383833 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:53 tscc-3-48 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:54 tscc-2-10 kernel: Lustre: 2974:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383787/real 1408383787]  req@ffff880b9e23e800 x1476519079928284/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-2-10 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:54 tscc-1-1 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383789/real 1408383789]  req@ffff881bdbcf6000 x1476519531649560/t0(0) o400->dolphin-MDT0000-mdc-ffff882026059c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-1-1 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:54 tscc-1-1 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:54 tscc-1-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026059c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:54 tscc-2-39 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383789/real 1408383789]  req@ffff8808214f5800 x1476547206138444/t0(0) o400->dolphin-MDT0000-mdc-ffff8808290f1c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-2-39 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:54 tscc-2-39 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808290f1c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:54 tscc-2-39 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:54 tscc-1-40 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383789/real 1408383789]  req@ffff882013f6fc00 x1476537020751008/t0(0) o400->dolphin-MDT0000-mdc-ffff881025cb7400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-0-62 kernel: Lustre: 2460:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383787/real 1408383787]  req@ffff8810343f4000 x1475809553542404/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-0-62 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:54 tscc-0-62 kernel: Lustre: dolphin-MDT0000-mdc-ffff881033ef6400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:54 tscc-1-11 kernel: Lustre: 2761:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383787/real 1408383787]  req@ffff8801df06bc00 x1476519742422636/t0(0) o400->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-1-11 kernel: Lustre: 2761:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:54 tscc-1-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026d94400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:54 tscc-1-11 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:54 tscc-2-52 kernel: Lustre: 2969:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383787/real 1408383787]  req@ffff88058ab15000 x1476290267066284/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-2-52 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:54 tscc-3-40 kernel: Lustre: 2735:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383790/real 1408383790]  req@ffff880e76998800 x1476520324363508/t0(0) o400->dolphin-MDT0000-mdc-ffff881025a8d400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-1-14 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383790/real 1408383790]  req@ffff88400fe5e000 x1476519630547384/t0(0) o400->dolphin-MDT0000-mdc-ffff884010213000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-2-52 kernel: Lustre: dolphin-MDT0000-mdc-ffff881067a76000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:54 tscc-0-20 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383790/real 1408383790]  req@ffff8810337b8400 x1476638723824920/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-0-20 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383790/real 1408383790]  req@ffff8810337b8c00 x1476638723824924/t0(0) o400->dolphin-MDT0000-mdc-ffff8810332bc400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-0-20 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:54 tscc-0-57 kernel: Lustre: 2240:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383787/real 1408383787]  req@ffff881033c53c00 x1475562021656572/t0(0) o400->dolphin-MDT0000-mdc-ffff881032828c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-0-57 kernel: Lustre: 2240:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 10:43:54 tscc-0-57 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:54 tscc-0-57 kernel: Lustre: dolphin-MDT0000-mdc-ffff881032828c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:54 tscc-gpu-7-7 kernel: Lustre: 2546:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383790/real 1408383790]  req@ffff880458cef400 x1476536280029948/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-gpu-7-7 kernel: Lustre: 2546:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:54 tscc-gpu-7-7 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:54 tscc-1-61 kernel: Lustre: 2742:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383790/real 1408383790]  req@ffff8807b5279400 x1476536929544780/t0(0) o400->dolphin-MDT0000-mdc-ffff88103115e400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-1-61 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:43:54 tscc-2-13 kernel: Lustre: 2971:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383790/real 1408383790]  req@ffff881e54a14800 x1476519055815068/t0(0) o400->dolphin-MDT0000-mdc-ffff881065172800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383834 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:54 tscc-2-13 kernel: Lustre: dolphin-MDT0000-mdc-ffff881065172800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:43:55 tscc-0-2 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383827/real 1408383827]  req@ffff880814c21c00 x1476547435068288/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383835 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:55 tscc-0-2 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:55 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383829/real 1408383829]  req@ffff8820267ac000 x1476536956386048/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383835 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:55 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:56 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383827/real 1408383827]  req@ffff882025a38000 x1476519511691588/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383836 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:56 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:56 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383827/real 1408383827]  req@ffff88015740e000 x1476547204759508/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383836 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:56 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:56 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383827/real 1408383827]  req@ffff88100a9dcc00 x1476519881774392/t0(0) o38->dolphin-MDT0000-mdc-ffff881025aa6800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383836 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:56 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:57 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383831/real 1408383831]  req@ffff880832268c00 x1476547271387056/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383837 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:57 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:57 tscc-1-35 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383831/real 1408383831]  req@ffff88159fbdbc00 x1476285713801872/t0(0) o38->dolphin-MDT0000-mdc-ffff881024c5a000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383837 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:57 tscc-1-35 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:57 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383831/real 1408383831]  req@ffff880d72eaa400 x1476537036478192/t0(0) o38->dolphin-MDT0000-mdc-ffff882026316400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383837 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:43:57 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:43:58 tscc-2-10 kernel: Lustre: 4711:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383822/real 1408383822]  req@ffff8803d6df9800 x1476519079930564/t0(0) o35->dolphin-MDT0000-mdc-ffff881066654400@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408383838 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 10:43:58 tscc-2-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff881066654400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:44:00 tscc-0-20 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff8810337b8800 x1476638723825192/t0(0) o38->dolphin-MDT0000-mdc-ffff8810332bc400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383840 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:00 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff880458cef800 x1476536280030220/t0(0) o38->dolphin-MDT0000-mdc-ffff880876dd2c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383840 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:00 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff8806ef0fa000 x1476536929545048/t0(0) o38->dolphin-MDT0000-mdc-ffff88103115e400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383840 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:00 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:00 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff882067fb2000 x1476519055815332/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383840 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:01 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff8818aab18c00 x1476519531650088/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383841 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:01 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 10:44:01 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff8808214f5000 x1476547206138972/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383841 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:01 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:01 tscc-1-40 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff882013f6f000 x1476537020751276/t0(0) o38->dolphin-MDT0000-mdc-ffff881025cb7400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383841 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:01 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383835/real 1408383835]  req@ffff880287b78800 x1476612876381396/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383841 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:03 tscc-1-45 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383832/real 1408383832]  req@ffff880d14a34400 x1476537015491964/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383843 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:03 tscc-1-45 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:03 tscc-0-62 kernel: Lustre: 2458:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff880816498000 x1475809553542932/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383843 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:03 tscc-0-62 kernel: Lustre: 2458:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:03 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff88068d4a6400 x1476519742423164/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383843 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:03 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:03 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff880167081800 x1476290267069060/t0(0) o38->dolphin-MDT0000-mdc-ffff881067a76000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383843 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:03 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:03 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff88103158cc00 x1475562021657100/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383843 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:03 tscc-2-10 kernel: Lustre: 2962:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff8803d6df9000 x1476519079930568/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383843 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:03 tscc-2-10 kernel: Lustre: 2962:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:05 tscc-1-40 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:44:05 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383845/real 1408383845]  req@ffff880e4ecaf800 x1476710234924296/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383851 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:05 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 10:44:08 tscc-3-56 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383804/real 1408383804]  req@ffff880737ec8800 x1476519829324532/t0(0) o400->dolphin-MDT0000-mdc-ffff8808317cc800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383848 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:08 tscc-3-56 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:08 tscc-gpu-5-4 kernel: Lustre: 2626:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383804/real 1408383804]  req@ffff880461fc5000 x1476612337738640/t0(0) o400->dolphin-MDT0000-mdc-ffff880877501800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383848 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:08 tscc-gpu-5-4 kernel: Lustre: 2626:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:09 tscc-0-31 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383805/real 1408383805]  req@ffff880812ae2400 x1476612857959548/t0(0) o400->dolphin-MDT0000-mdc-ffff880830a6b000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383849 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:09 tscc-0-31 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:09 tscc-1-32 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383804/real 1408383804]  req@ffff881d3d079800 x1476439898236900/t0(0) o400->dolphin-MDT0000-mdc-ffff8810258fb000@172.25.32.53@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383848 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:09 tscc-1-32 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:09 tscc-0-28 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383805/real 1408383805]  req@ffff880ff7490c00 x1476547206477512/t0(0) o400->dolphin-MDT0000-mdc-ffff880831447800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383849 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:09 tscc-0-28 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:12 tscc-1-48 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383808/real 1408383808]  req@ffff8810273b8400 x1476537001864288/t0(0) o400->dolphin-MDT0000-mdc-ffff882028f17800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383852 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:12 tscc-1-48 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:12 tscc-1-45 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383808/real 1408383808]  req@ffff880d2ad08400 x1476537015491700/t0(0) o400->dolphin-MDT0000-mdc-ffff882028c86c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383852 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:12 tscc-0-10 kernel: Lustre: 2763:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383808/real 1408383808]  req@ffff880823992800 x1476547227438128/t0(0) o400->dolphin-MDT0000-mdc-ffff8810322ef000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383852 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:12 tscc-0-10 kernel: Lustre: 2763:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:12 tscc-1-6 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383805/real 1408383805]  req@ffff881e0c012800 x1476519511691328/t0(0) o400->dolphin-MDT0000-mdc-ffff8810266d7400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383852 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:12 tscc-1-6 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:12 tscc-3-43 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383808/real 1408383808]  req@ffff88050243fc00 x1476519948866352/t0(0) o400->dolphin-MDT0000-mdc-ffff88101eab6000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383852 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:12 tscc-3-43 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:12 tscc-0-36 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383805/real 1408383805]  req@ffff880827a82800 x1476547204759244/t0(0) o400->dolphin-MDT0000-mdc-ffff880831f26c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383852 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:12 tscc-0-36 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:12 tscc-3-48 kernel: Lustre: 2760:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383808/real 1408383808]  req@ffff8816ee7cd800 x1476287317244584/t0(0) o400->dolphin-MDT0000-mdc-ffff8810266f7400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383852 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:12 tscc-3-48 kernel: Lustre: 2760:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:12 tscc-3-51 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383805/real 1408383805]  req@ffff8801959e3000 x1476519881774128/t0(0) o400->dolphin-MDT0000-mdc-ffff881025aa6800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383852 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:12 tscc-3-51 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:12 tscc-0-2 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383806/real 1408383806]  req@ffff880810b08400 x1476547435068024/t0(0) o400->dolphin-MDT0000-mdc-ffff88083175c400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383852 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:12 tscc-0-2 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:13 tscc-2-10 kernel: Lustre: 2962:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383838/real 1408383838]  req@ffff880b9e23e800 x1476519079930572/t0(0) o38->dolphin-MDT0000-mdc-ffff881066654400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383853 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:14 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383848/real 1408383848]  req@ffff880708fe1c00 x1476519829324800/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383854 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:14 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383848/real 1408383848]  req@ffff881e45db9c00 x1476439898237168/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383854 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:14 tscc-3-40 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383790/real 1408383790]  req@ffff880cb5bd7800 x1476520324363504/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383854 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:14 tscc-3-40 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:14 tscc-3-40 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:44:14 tscc-1-56 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383810/real 1408383810]  req@ffff882013b98000 x1476536956385784/t0(0) o400->dolphin-MDT0000-mdc-ffff882028ff0000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383854 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:14 tscc-1-56 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:15 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383849/real 1408383849]  req@ffff880812ae2800 x1476612857959816/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383855 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:15 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383849/real 1408383849]  req@ffff88102fd2f800 x1476547206477784/t0(0) o38->dolphin-MDT0000-mdc-ffff880831447800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383855 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:16 tscc-0-7 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383812/real 1408383812]  req@ffff8808297d6000 x1476547271386792/t0(0) o400->dolphin-MDT0000-mdc-ffff880833252000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383856 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:16 tscc-0-7 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:16 tscc-1-35 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383812/real 1408383812]  req@ffff881254defc00 x1476285713801612/t0(0) o400->dolphin-MDT0000-mdc-ffff881024c5a000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383856 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:16 tscc-1-35 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:16 tscc-1-53 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383812/real 1408383812]  req@ffff880d54d52000 x1476537036477932/t0(0) o400->dolphin-MDT0000-mdc-ffff882026316400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383856 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:16 tscc-1-53 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:18 tscc-0-10 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383852/real 1408383852]  req@ffff880826d72000 x1476547227438396/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383858 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:18 tscc-1-14 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383790/real 1408383790]  req@ffff8840000f7800 x1476519630547380/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408383858 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:18 tscc-1-14 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:18 tscc-1-14 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:44:19 tscc-1-1 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383814/real 1408383814]  req@ffff881bdbcf6800 x1476519531649824/t0(0) o400->dolphin-MDT0000-mdc-ffff882026059c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383859 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:19 tscc-1-1 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:19 tscc-2-39 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383814/real 1408383814]  req@ffff8804de845c00 x1476547206138708/t0(0) o400->dolphin-MDT0000-mdc-ffff8808290f1c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383859 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:19 tscc-2-39 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:19 tscc-1-11 kernel: Lustre: 2768:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383812/real 1408383812]  req@ffff8802b323cc00 x1476519742422900/t0(0) o400->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383859 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:19 tscc-1-11 kernel: Lustre: 2768:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:19 tscc-0-57 kernel: Lustre: 2246:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383812/real 1408383812]  req@ffff880f1b5d1400 x1475562021656836/t0(0) o400->dolphin-MDT0000-mdc-ffff881032828c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383859 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:19 tscc-0-57 kernel: Lustre: 2246:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:20 tscc-0-62 kernel: Lustre: 2465:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383812/real 1408383812]  req@ffff88076cd1f000 x1475809553542672/t0(0) o400->dolphin-MDT0000-mdc-ffff881033ef6400@172.25.32.53@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383859 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:20 tscc-0-62 kernel: Lustre: 2465:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:20 tscc-0-2 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383852/real 1408383852]  req@ffff880815765c00 x1476547435068296/t0(0) o38->dolphin-MDT0000-mdc-ffff88083175c400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383860 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:20 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383854/real 1408383854]  req@ffff882013bb9c00 x1476536956386052/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383860 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:21 tscc-2-52 kernel: Lustre: 2971:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383812/real 1408383812]  req@ffff88066810ac00 x1476290267067748/t0(0) o400->dolphin-MDT0000-mdc-ffff881067a76000@172.25.32.53@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383859 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:21 tscc-2-52 kernel: Lustre: 2971:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:21 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383852/real 1408383852]  req@ffff881d7979d400 x1476519511691596/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383861 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:21 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383852/real 1408383852]  req@ffff880827a82000 x1476547204759516/t0(0) o38->dolphin-MDT0000-mdc-ffff880831f26c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383861 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:21 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383852/real 1408383852]  req@ffff88100a9dc000 x1476519881774400/t0(0) o38->dolphin-MDT0000-mdc-ffff881025aa6800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383861 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:22 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383856/real 1408383856]  req@ffff880819fba000 x1476547271387064/t0(0) o38->dolphin-MDT0000-mdc-ffff880833252000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383862 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:22 tscc-1-35 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383856/real 1408383856]  req@ffff88138dd3ac00 x1476285713801884/t0(0) o38->dolphin-MDT0000-mdc-ffff881024c5a000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383862 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:22 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383856/real 1408383856]  req@ffff880fb514dc00 x1476537036478204/t0(0) o38->dolphin-MDT0000-mdc-ffff882026316400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383862 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:23 tscc-1-40 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383845/real 1408383845]  req@ffff882013f6f000 x1476537020751536/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383863 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:23 tscc-1-40 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:25 tscc-2-10 kernel: Lustre: 2972:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383812/real 1408383812]  req@ffff8806392e4400 x1476519079929808/t0(0) o400->dolphin-MDT0000-mdc-ffff881066654400@172.25.32.53@tcp:12/10 lens 224/224 e 0 to 1 dl 1408383865 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:26 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383859/real 1408383859]  req@ffff881931ef3000 x1476519531650096/t0(0) o38->dolphin-MDT0000-mdc-ffff882026059c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383866 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:26 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383859/real 1408383859]  req@ffff8808214f5000 x1476547206138980/t0(0) o38->dolphin-MDT0000-mdc-ffff8808290f1c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383866 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:28 tscc-0-62 kernel: Lustre: 2458:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383859/real 1408383859]  req@ffff88082eeffc00 x1475809553542940/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383868 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:28 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383859/real 1408383859]  req@ffff88102879a400 x1476519742423172/t0(0) o38->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383868 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:28 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383859/real 1408383859]  req@ffff88066810a800 x1476290267070272/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383868 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:28 tscc-0-20 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff8810184bcc00 x1476638723825452/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383868 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:28 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383859/real 1408383859]  req@ffff881034358000 x1475562021657108/t0(0) o38->dolphin-MDT0000-mdc-ffff881032828c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383868 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:30 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383859/real 1408383859]  req@ffff880458cef800 x1476536280030500/t0(0) o38->dolphin-MDT0000-mdc-ffff880876dd2c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383870 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:30 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383859/real 1408383859]  req@ffff880778756400 x1476536929545312/t0(0) o38->dolphin-MDT0000-mdc-ffff88103115e400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383870 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:30 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383859/real 1408383859]  req@ffff881ccbb15800 x1476519055815612/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383870 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:40 tscc-3-40 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383854/real 1408383854]  req@ffff880edbbf8c00 x1476520324364036/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383880 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:41 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383870/real 1408383870]  req@ffff880e4b810800 x1476710234924564/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383881 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:41 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:42 tscc-1-40 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383870/real 1408383870]  req@ffff882013f6f400 x1476537020751544/t0(0) o38->dolphin-MDT0000-mdc-ffff881025cb7400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383882 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:42 tscc-1-40 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:42 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383834/real 1408383834]  req@ffff8820681f4c00 x1476519055815596/t0(0) o38->dolphin-MDT0000-mdc-ffff881065172800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383882 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:43 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383860/real 1408383860]  req@ffff8806af255000 x1476612876381664/t0(0) o38->dolphin-MDT0000-mdc-ffff8808300d3800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383883 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:43 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 10:44:44 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383873/real 1408383873]  req@ffff88077fa2a400 x1476519829325064/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383884 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:44 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:44 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383873/real 1408383873]  req@ffff881d3d079400 x1476439898237432/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383884 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:44 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:44 tscc-gpu-5-4 kernel: Lustre: 2620:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383873/real 1408383873]  req@ffff88046356b800 x1476612337739172/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383884 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:44 tscc-gpu-5-4 kernel: Lustre: 2620:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 10:44:44 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383884/real 1408383884]  req@ffff88044b6aac00 x1476536280030760/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383923 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:44 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:44 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383884/real 1408383884]  req@ffff8808147f7000 x1476536929545572/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:44 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:45 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383874/real 1408383874]  req@ffff880812efec00 x1476612857960080/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383885 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:45 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:45 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383874/real 1408383874]  req@ffff88102fd2fc00 x1476547206478048/t0(0) o38->dolphin-MDT0000-mdc-ffff880831447800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383885 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:45 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:48 tscc-1-48 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383877/real 1408383877]  req@ffff880cfd20b800 x1476537001864824/t0(0) o38->dolphin-MDT0000-mdc-ffff882028f17800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383888 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:48 tscc-1-48 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 10:44:48 tscc-0-10 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383877/real 1408383877]  req@ffff880826789c00 x1476547227438660/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383888 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:48 tscc-0-10 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:48 tscc-3-43 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383877/real 1408383877]  req@ffff880ff49b3400 x1476519948866884/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383888 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:48 tscc-3-43 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 10:44:48 tscc-1-14 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383858/real 1408383858]  req@ffff884010853c00 x1476519630547912/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383888 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:48 tscc-3-48 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383877/real 1408383877]  req@ffff881bec497400 x1476287317245120/t0(0) o38->dolphin-MDT0000-mdc-ffff8810266f7400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383888 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:48 tscc-3-48 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 10:44:50 tscc-0-2 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383877/real 1408383877]  req@ffff88081741fc00 x1476547435068556/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:50 tscc-0-2 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:50 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383879/real 1408383879]  req@ffff882013bb9000 x1476536956386316/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:50 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:51 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383877/real 1408383877]  req@ffff882027423c00 x1476519511691860/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383891 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:51 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:51 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383877/real 1408383877]  req@ffff880832b32800 x1476547204759780/t0(0) o38->dolphin-MDT0000-mdc-ffff880831f26c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383891 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:51 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:51 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383877/real 1408383877]  req@ffff88100a9dc000 x1476519881774664/t0(0) o38->dolphin-MDT0000-mdc-ffff881025aa6800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383891 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:51 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:52 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383881/real 1408383881]  req@ffff880819fba000 x1476547271387324/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383892 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:52 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:52 tscc-1-35 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383881/real 1408383881]  req@ffff88138dd3a400 x1476285713802144/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383892 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:52 tscc-1-35 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:52 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383881/real 1408383881]  req@ffff880d5635dc00 x1476537036478464/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383892 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:52 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:53 tscc-1-45 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383882/real 1408383882]  req@ffff88102875e400 x1476537015492236/t0(0) o38->dolphin-MDT0000-mdc-ffff882028c86c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383893 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:53 tscc-1-45 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 10:44:56 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383884/real 1408383884]  req@ffff881cdbdb5400 x1476519531650360/t0(0) o38->dolphin-MDT0000-mdc-ffff882026059c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:56 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:56 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383884/real 1408383884]  req@ffff8808214f5c00 x1476547206139240/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:56 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:58 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383898/real 1408383898]  req@ffff88070b5c3000 x1476519829325328/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383909 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:58 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:58 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383898/real 1408383898]  req@ffff881e45db9c00 x1476439898237696/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383909 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:58 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:58 tscc-0-62 kernel: Lustre: 2458:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383884/real 1408383884]  req@ffff880756a3c800 x1475809553543208/t0(0) o38->dolphin-MDT0000-mdc-ffff881033ef6400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383898 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:58 tscc-0-62 kernel: Lustre: 2458:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:58 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383884/real 1408383884]  req@ffff88102879a400 x1476519742423436/t0(0) o38->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383898 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:58 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:58 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383884/real 1408383884]  req@ffff88023bd23c00 x1476290267071732/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383898 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:58 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:58 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383884/real 1408383884]  req@ffff881031524000 x1475562021657372/t0(0) o38->dolphin-MDT0000-mdc-ffff881032828c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383898 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:58 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:59 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383899/real 1408383899]  req@ffff880812e01c00 x1476612857960344/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383910 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:59 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:44:59 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383899/real 1408383899]  req@ffff88102fd2fc00 x1476547206478308/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383910 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:44:59 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:02 tscc-0-10 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383902/real 1408383902]  req@ffff880826d72000 x1476547227438924/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383913 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:02 tscc-0-10 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:02 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383902/real 1408383902]  req@ffff882025c02400 x1476519511692124/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383916 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:02 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:02 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383902/real 1408383902]  req@ffff88042d283c00 x1476547204760040/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383916 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:02 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:02 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383902/real 1408383902]  req@ffff88100a9dcc00 x1476519881774924/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383916 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:02 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:02 tscc-2-10 kernel: Lustre: 2962:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383888/real 1408383888]  req@ffff8806392e4400 x1476519079931088/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383902 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:02 tscc-2-10 kernel: Lustre: 2962:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 10:45:02 tscc-0-2 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383902/real 1408383902]  req@ffff88081741f000 x1476547435068820/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383915 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:02 tscc-0-2 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:04 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383904/real 1408383904]  req@ffff882013bb9c00 x1476536956386580/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383915 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:04 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:06 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383906/real 1408383906]  req@ffff880819fba000 x1476547271387588/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383917 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:06 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:06 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383906/real 1408383906]  req@ffff880f6147ec00 x1476537036478728/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383917 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:06 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:07 tscc-1-40 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383895/real 1408383895]  req@ffff881025d40400 x1476537020751808/t0(0) o38->dolphin-MDT0000-mdc-ffff881025cb7400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408383907 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:09 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383909/real 1408383909]  req@ffff881cdbcc2c00 x1476519531650620/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383921 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:09 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:09 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383909/real 1408383909]  req@ffff8808214f5800 x1476547206139504/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383921 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:09 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:09 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383909/real 1408383909]  req@ffff880fb01de800 x1476519742423696/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383923 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:09 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:09 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383909/real 1408383909]  req@ffff880d7e413800 x1476290267073192/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383923 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:09 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:09 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408383909/real 1408383909]  req@ffff880c3c0c9800 x1475562021657632/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383923 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:09 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 10:45:15 tscc-gpu-7-7 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 10:45:15 tscc-gpu-7-7 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:15 tscc-1-40 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 10:45:15 tscc-0-20 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 10:45:15 tscc-0-20 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:15 tscc-0-15 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 10:45:15 tscc-1-61 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 10:45:15 tscc-1-61 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:15 tscc-2-13 kernel: Lustre: dolphin-MDT0000-mdc-ffff881065172800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:15 tscc-2-13 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:20 tscc-1-40 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025cb7400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:20 tscc-login1 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 10:45:23 tscc-3-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808317cc800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:23 tscc-3-56 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:23 tscc-1-32 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810258fb000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:23 tscc-1-32 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:23 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:24 tscc-0-31 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830a6b000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:24 tscc-0-31 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:24 tscc-0-28 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831447800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:24 tscc-0-28 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:28 tscc-3-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025aa6800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:28 tscc-3-51 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:28 tscc-1-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810266d7400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:28 tscc-0-10 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 10:45:28 tscc-0-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff88083175c400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:28 tscc-3-43 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101eab6000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:28 tscc-3-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810266f7400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:28 tscc-0-36 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831f26c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:28 tscc-0-36 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:28 tscc-1-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028f17800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:29 tscc-3-40 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025a8d400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:29 tscc-3-40 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:29 tscc-1-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028ff0000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:30 tscc-3-40 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383904/real 1408383904]  req@ffff8810107f5400 x1476520324364300/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383930 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:30 tscc-3-40 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 10:45:31 tscc-0-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880833252000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:31 tscc-1-35 kernel: Lustre: dolphin-MDT0000-mdc-ffff881024c5a000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:31 tscc-1-53 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026316400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:32 tscc-1-45 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028c86c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:33 tscc-1-14 kernel: Lustre: dolphin-MDT0000-mdc-ffff884010213000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:33 tscc-1-14 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:34 tscc-1-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026059c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:34 tscc-1-1 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:34 tscc-2-39 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808290f1c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:34 tscc-2-39 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:34 tscc-0-62 kernel: Lustre: dolphin-MDT0000-mdc-ffff881033ef6400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:34 tscc-1-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026d94400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:34 tscc-1-11 kernel: Lustre: Skipped 1 previous similar message
Aug 18 10:45:34 tscc-2-52 kernel: Lustre: dolphin-MDT0000-mdc-ffff881067a76000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:34 tscc-0-57 kernel: Lustre: dolphin-MDT0000-mdc-ffff881032828c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:38 tscc-1-14 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408383908/real 1408383908]  req@ffff883ff0f32c00 x1476519630548176/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408383938 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:45:38 tscc-1-14 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 10:45:43 tscc-2-10 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 10:45:43 tscc-3-51 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 10:45:45 tscc-login1 kernel: Lustre: Evicted from dolphin-MDT0000_UUID (at 172.25.32.25@tcp) after server handle changed from 0x30845f5536d15fa9 to 0x30845f5537760407
Aug 18 10:45:45 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 10:45:45 tscc-login1 kernel: LustreError: 19271:0:(file.c:171:ll_close_inode_openhandle()) inode 144115494327241590 mdc close failed: rc = -5
Aug 18 10:45:45 tscc-login1 kernel: LustreError: 39086:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 10:45:45 tscc-login1 kernel: LustreError: 39086:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 8 previous similar messages
Aug 18 10:45:45 tscc-login1 kernel: LustreError: 39086:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x20000512f:0x4c0:0x0] error -5.
Aug 18 10:45:45 tscc-login1 kernel: LustreError: 39086:0:(vvp_io.c:1229:vvp_io_init()) Skipped 88 previous similar messages
Aug 18 10:45:45 tscc-login1 kernel: LustreError: 19271:0:(file.c:171:ll_close_inode_openhandle()) Skipped 6 previous similar messages
Aug 18 10:45:45 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:45:54 tscc-3-40 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 10:45:58 tscc-1-14 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 10:47:48 tscc-0-23 kernel: Lustre: 53478:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408384037/real 1408384037]  req@ffff880cd42f9400 x1476547160031516/t0(0) o35->dolphin-MDT0000-mdc-ffff881030c9b000@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408384068 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 10:47:48 tscc-0-23 kernel: Lustre: dolphin-MDT0000-mdc-ffff881030c9b000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 10:48:18 tscc-0-23 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408384038/real 1408384038]  req@ffff880bf8183800 x1476547160031520/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408384098 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:48:18 tscc-0-23 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 10:48:20 tscc-0-23 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408384068/real 1408384068]  req@ffff88101111d800 x1476547160032052/t0(0) o38->dolphin-MDT0000-mdc-ffff881030c9b000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408384100 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:48:25 tscc-0-23 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 10:49:35 tscc-0-23 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408384123/real 1408384123]  req@ffff8810311dc800 x1476547160032372/t0(0) o38->dolphin-MDT0000-mdc-ffff881030c9b000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408384175 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:49:58 tscc-0-23 kernel: Lustre: dolphin-MDT0000-mdc-ffff881030c9b000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 10:52:15 tscc-0-63 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 10:52:20 tscc-0-63 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408384335/real 1408384335]  req@ffff881032063c00 x1476798012456964/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408384340 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:52:21 tscc-0-63 kernel: LustreError: 2655:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff881032063800 x1476798012456968/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 10:52:30 tscc-0-63 kernel: LustreError: 2756:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff881032063000 x1476798012456976/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 10:52:45 tscc-0-63 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408384360/real 1408384360]  req@ffff881032063c00 x1476798012457060/t0(0) o38->dolphin-MDT0000-mdc-ffff881031346c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408384365 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:53:05 tscc-0-63 kernel: Lustre: Layout lock feature supported.
Aug 18 10:53:06 tscc-0-63 kernel: Lustre: Mounted dolphin-client
Aug 18 10:53:17 tscc-gpu-3-1 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 10:53:23 tscc-gpu-3-1 kernel: Lustre: 2906:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408384398/real 1408384398]  req@ffff880876440c00 x1476798077468676/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408384403 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:53:24 tscc-gpu-3-1 kernel: LustreError: 2842:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880876440800 x1476798077468680/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 10:53:33 tscc-gpu-3-1 kernel: LustreError: 2936:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880476190c00 x1476798077468688/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 10:53:48 tscc-gpu-3-1 kernel: Lustre: 2906:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408384423/real 1408384423]  req@ffff880876440800 x1476798077468772/t0(0) o38->dolphin-MDT0000-mdc-ffff880878f15400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408384428 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:54:08 tscc-gpu-3-1 kernel: Lustre: Layout lock feature supported.
Aug 18 10:54:08 tscc-gpu-3-1 kernel: Lustre: Mounted dolphin-client
Aug 18 10:54:25 tscc-gpu-3-2 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 10:54:30 tscc-gpu-3-2 kernel: Lustre: 2905:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408384465/real 1408384465]  req@ffff88087a644c00 x1476798148771844/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408384470 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:54:31 tscc-gpu-3-2 kernel: LustreError: 2841:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88087a644800 x1476798148771848/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 10:54:38 tscc-gpu-3-2 kernel: LustreError: 2935:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88087a644000 x1476798148771856/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 10:54:55 tscc-gpu-3-2 kernel: Lustre: 2905:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408384490/real 1408384490]  req@ffff8808766d9c00 x1476798148771940/t0(0) o38->dolphin-MDT0000-mdc-ffff8808767dd000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408384495 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 10:55:19 tscc-gpu-3-2 kernel: Lustre: Layout lock feature supported.
Aug 18 10:55:19 tscc-gpu-3-2 kernel: Lustre: Mounted dolphin-client
Aug 18 11:04:25 tscc-login1 kernel: Lustre: 56313:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385058/real 1408385058]  req@ffff88078acdf400 x1476710234967304/t0(0) o101->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.25@tcp:12/10 lens 600/2536 e 0 to 1 dl 1408385065 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 11:04:25 tscc-login1 kernel: Lustre: 56313:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Aug 18 11:04:25 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:25 tscc-1-14 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385058/real 1408385058]  req@ffff88400fe5e400 x1476519630560352/t0(0) o400->dolphin-MDT0000-mdc-ffff884010213000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385065 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:25 tscc-1-14 kernel: Lustre: dolphin-MDT0000-mdc-ffff884010213000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:26 tscc-1-1 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385059/real 1408385059]  req@ffff881931ef3c00 x1476519531662800/t0(0) o400->dolphin-MDT0000-mdc-ffff882026059c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385066 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:26 tscc-1-1 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:26 tscc-1-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026059c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:26 tscc-2-39 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385059/real 1408385059]  req@ffff88080e067000 x1476547206151684/t0(0) o400->dolphin-MDT0000-mdc-ffff8808290f1c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385066 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:26 tscc-2-39 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:26 tscc-2-39 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808290f1c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:26 tscc-0-62 kernel: Lustre: 2465:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385059/real 1408385059]  req@ffff8810308ffc00 x1475809553555640/t0(0) o400->dolphin-MDT0000-mdc-ffff881033ef6400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385066 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:26 tscc-0-62 kernel: Lustre: 2465:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 11:04:26 tscc-0-62 kernel: Lustre: dolphin-MDT0000-mdc-ffff881033ef6400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:26 tscc-1-11 kernel: Lustre: 2757:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385059/real 1408385059]  req@ffff881014160c00 x1476519742436860/t0(0) o400->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385066 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:26 tscc-1-11 kernel: Lustre: 2757:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:26 tscc-1-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026d94400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:26 tscc-0-20 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385059/real 1408385059]  req@ffff8810337b8800 x1476638723838180/t0(0) o400->dolphin-MDT0000-mdc-ffff8810332bc400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385066 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:26 tscc-0-20 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 11:04:26 tscc-0-20 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810332bc400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:26 tscc-0-57 kernel: Lustre: 2246:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385059/real 1408385059]  req@ffff880dc0e27400 x1475562021669804/t0(0) o400->dolphin-MDT0000-mdc-ffff881032828c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385066 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:26 tscc-0-57 kernel: Lustre: 2246:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:26 tscc-0-57 kernel: Lustre: dolphin-MDT0000-mdc-ffff881032828c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:26 tscc-gpu-7-7 kernel: Lustre: 2542:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385059/real 1408385059]  req@ffff880477571800 x1476536280043204/t0(0) o400->dolphin-MDT0000-mdc-ffff880876dd2c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385066 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:26 tscc-gpu-7-7 kernel: Lustre: 2542:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:26 tscc-gpu-7-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876dd2c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:26 tscc-1-61 kernel: Lustre: 2739:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385059/real 1408385059]  req@ffff8806d2648800 x1476536929558020/t0(0) o400->dolphin-MDT0000-mdc-ffff88103115e400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385066 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:26 tscc-1-61 kernel: Lustre: 2739:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:26 tscc-1-61 kernel: Lustre: dolphin-MDT0000-mdc-ffff88103115e400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:26 tscc-gpu-5-4 kernel: Lustre: 27381:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385059/real 1408385059]  req@ffff88084cc50800 x1476612337751876/t0(0) o101->dolphin-MDT0000-mdc-ffff880877501800@172.25.32.25@tcp:12/10 lens 584/2536 e 0 to 1 dl 1408385066 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 11:04:26 tscc-gpu-5-4 kernel: Lustre: 27381:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 11:04:26 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:26 tscc-2-13 kernel: Lustre: 2970:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385059/real 1408385059]  req@ffff880a63c6cc00 x1476519055828316/t0(0) o400->dolphin-MDT0000-mdc-ffff881065172800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385066 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:26 tscc-2-13 kernel: Lustre: 2970:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:04:26 tscc-2-13 kernel: Lustre: dolphin-MDT0000-mdc-ffff881065172800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:27 tscc-2-52 kernel: Lustre: 2967:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385059/real 1408385059]  req@ffff880ead31dc00 x1476290267140236/t0(0) o400->dolphin-MDT0000-mdc-ffff881067a76000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385066 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:27 tscc-2-52 kernel: Lustre: 2967:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:27 tscc-2-52 kernel: Lustre: dolphin-MDT0000-mdc-ffff881067a76000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:27 tscc-0-15 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385060/real 1408385060]  req@ffff880830b29000 x1476612877231856/t0(0) o400->dolphin-MDT0000-mdc-ffff8808300d3800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385067 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:27 tscc-0-15 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:04:27 tscc-0-15 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808300d3800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:30 tscc-2-10 kernel: Lustre: 2969:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385063/real 1408385063]  req@ffff8808eb086000 x1476519079943528/t0(0) o400->dolphin-MDT0000-mdc-ffff881066654400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385070 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:30 tscc-2-10 kernel: Lustre: 2969:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 11:04:30 tscc-2-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff881066654400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:32 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385066/real 1408385066]  req@ffff8818f4dff000 x1476519531663060/t0(0) o38->dolphin-MDT0000-mdc-ffff882026059c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385072 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:32 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385066/real 1408385066]  req@ffff880396b51400 x1476547206151944/t0(0) o38->dolphin-MDT0000-mdc-ffff8808290f1c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385072 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:32 tscc-0-62 kernel: Lustre: 2458:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385066/real 1408385066]  req@ffff88103094d800 x1475809553555900/t0(0) o38->dolphin-MDT0000-mdc-ffff881033ef6400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385072 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:32 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385066/real 1408385066]  req@ffff880fddf23c00 x1476519742437120/t0(0) o38->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385072 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:32 tscc-0-20 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385066/real 1408385066]  req@ffff881028d8bc00 x1476638723838440/t0(0) o38->dolphin-MDT0000-mdc-ffff8810332bc400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385072 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:32 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385066/real 1408385066]  req@ffff881032c26400 x1475562021670064/t0(0) o38->dolphin-MDT0000-mdc-ffff881032828c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385072 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:32 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385066/real 1408385066]  req@ffff88045795b400 x1476536280043464/t0(0) o38->dolphin-MDT0000-mdc-ffff880876dd2c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385072 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:32 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385066/real 1408385066]  req@ffff8808155b2000 x1476536929558280/t0(0) o38->dolphin-MDT0000-mdc-ffff88103115e400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385072 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:32 tscc-gpu-5-4 kernel: Lustre: 2620:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385066/real 1408385066]  req@ffff88084cc50400 x1476612337751880/t0(0) o38->dolphin-MDT0000-mdc-ffff880877501800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385072 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:32 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385066/real 1408385066]  req@ffff880123f40400 x1476519055828576/t0(0) o38->dolphin-MDT0000-mdc-ffff881065172800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385072 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:33 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385067/real 1408385067]  req@ffff880f09325000 x1476290267140848/t0(0) o38->dolphin-MDT0000-mdc-ffff881067a76000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385073 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:34 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385065/real 1408385065]  req@ffff88082477fc00 x1476710234967440/t0(0) o38->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385074 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:34 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:04:36 tscc-2-10 kernel: Lustre: 2962:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385070/real 1408385070]  req@ffff880347ab3400 x1476519079943788/t0(0) o38->dolphin-MDT0000-mdc-ffff881066654400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:37 tscc-1-40 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385070/real 1408385070]  req@ffff880f2309fc00 x1476537020764248/t0(0) o400->dolphin-MDT0000-mdc-ffff881025cb7400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:37 tscc-1-40 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025cb7400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:39 tscc-gpu-3-1 kernel: Lustre: 2907:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385072/real 1408385072]  req@ffff880878a7d800 x1476798077475396/t0(0) o400->dolphin-MDT0000-mdc-ffff880878f15400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385079 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:39 tscc-gpu-3-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880878f15400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:40 tscc-3-56 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385073/real 1408385073]  req@ffff88077fa2ac00 x1476519829337772/t0(0) o400->dolphin-MDT0000-mdc-ffff8808317cc800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385080 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:40 tscc-3-56 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:40 tscc-3-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808317cc800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:40 tscc-1-32 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385073/real 1408385073]  req@ffff881db3d6b800 x1476439898250140/t0(0) o400->dolphin-MDT0000-mdc-ffff8810258fb000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385080 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:40 tscc-1-32 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:40 tscc-1-32 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810258fb000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:41 tscc-0-31 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385074/real 1408385074]  req@ffff8804c14c8800 x1476612857972788/t0(0) o400->dolphin-MDT0000-mdc-ffff880830a6b000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385081 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:41 tscc-0-31 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:41 tscc-0-31 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830a6b000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:41 tscc-0-28 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385074/real 1408385074]  req@ffff881011822c00 x1476547206490752/t0(0) o400->dolphin-MDT0000-mdc-ffff880831447800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385081 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:41 tscc-0-28 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:41 tscc-0-28 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831447800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:43 tscc-1-40 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385077/real 1408385077]  req@ffff8810251e2800 x1476537020764508/t0(0) o38->dolphin-MDT0000-mdc-ffff881025cb7400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385083 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:44 tscc-1-48 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385077/real 1408385077]  req@ffff880eb0c93400 x1476537001877528/t0(0) o400->dolphin-MDT0000-mdc-ffff882028f17800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385084 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:44 tscc-1-48 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 11:04:44 tscc-1-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028f17800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:44 tscc-0-10 kernel: Lustre: 2754:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385077/real 1408385077]  req@ffff880826789800 x1476547227451368/t0(0) o400->dolphin-MDT0000-mdc-ffff8810322ef000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385084 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:44 tscc-0-10 kernel: Lustre: 2754:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:44 tscc-0-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810322ef000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:44 tscc-1-6 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385077/real 1408385077]  req@ffff881a776e9000 x1476519511704568/t0(0) o400->dolphin-MDT0000-mdc-ffff8810266d7400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385084 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:44 tscc-1-6 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:44 tscc-1-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810266d7400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:44 tscc-3-43 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385077/real 1408385077]  req@ffff880be8bc2800 x1476519948879592/t0(0) o400->dolphin-MDT0000-mdc-ffff88101eab6000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385084 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:44 tscc-3-43 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 11:04:44 tscc-3-43 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101eab6000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:44 tscc-0-36 kernel: Lustre: 2735:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385077/real 1408385077]  req@ffff880121a78400 x1476547204772480/t0(0) o400->dolphin-MDT0000-mdc-ffff880831f26c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385084 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:44 tscc-0-36 kernel: Lustre: 2735:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:44 tscc-0-36 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831f26c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:44 tscc-3-48 kernel: Lustre: 2751:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385077/real 1408385077]  req@ffff881bec497000 x1476287317257824/t0(0) o400->dolphin-MDT0000-mdc-ffff8810266f7400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385084 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:44 tscc-3-48 kernel: Lustre: 2751:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 11:04:44 tscc-3-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810266f7400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:44 tscc-3-51 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385077/real 1408385077]  req@ffff88100a9dcc00 x1476519881787368/t0(0) o400->dolphin-MDT0000-mdc-ffff881025aa6800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385084 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:44 tscc-3-51 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:44 tscc-3-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025aa6800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:44 tscc-0-2 kernel: Lustre: 2751:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385077/real 1408385077]  req@ffff88082f8cb800 x1476547435081264/t0(0) o400->dolphin-MDT0000-mdc-ffff88083175c400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385084 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:44 tscc-0-2 kernel: Lustre: 2751:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:44 tscc-0-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff88083175c400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:45 tscc-gpu-3-1 kernel: Lustre: 2906:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385079/real 1408385079]  req@ffff88085fb39800 x1476798077475656/t0(0) o38->dolphin-MDT0000-mdc-ffff880878f15400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385085 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:46 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385080/real 1408385080]  req@ffff880708fe1c00 x1476519829338032/t0(0) o38->dolphin-MDT0000-mdc-ffff8808317cc800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385086 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:46 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385080/real 1408385080]  req@ffff881dde29dc00 x1476439898250400/t0(0) o38->dolphin-MDT0000-mdc-ffff8810258fb000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385086 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:46 tscc-3-40 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385079/real 1408385079]  req@ffff880cb5bd7800 x1476520324376740/t0(0) o400->dolphin-MDT0000-mdc-ffff881025a8d400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385086 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:46 tscc-3-40 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025a8d400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:46 tscc-1-56 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385079/real 1408385079]  req@ffff88101b23a400 x1476536956399024/t0(0) o400->dolphin-MDT0000-mdc-ffff882028ff0000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385086 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:46 tscc-1-56 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:46 tscc-1-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028ff0000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:47 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385081/real 1408385081]  req@ffff8807fb89b400 x1476612857973048/t0(0) o38->dolphin-MDT0000-mdc-ffff880830a6b000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385087 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:47 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385081/real 1408385081]  req@ffff880ff3543000 x1476547206491012/t0(0) o38->dolphin-MDT0000-mdc-ffff880831447800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385087 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:48 tscc-0-7 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385081/real 1408385081]  req@ffff88081ca91c00 x1476547271400032/t0(0) o400->dolphin-MDT0000-mdc-ffff880833252000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385088 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:48 tscc-0-7 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:48 tscc-0-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880833252000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:48 tscc-1-35 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385081/real 1408385081]  req@ffff880f9be7f000 x1476285713814852/t0(0) o400->dolphin-MDT0000-mdc-ffff881024c5a000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385088 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:48 tscc-1-35 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 11:04:48 tscc-1-35 kernel: Lustre: dolphin-MDT0000-mdc-ffff881024c5a000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:48 tscc-1-53 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385081/real 1408385081]  req@ffff880f83acbc00 x1476537036491172/t0(0) o400->dolphin-MDT0000-mdc-ffff882026316400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385088 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:48 tscc-1-53 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:48 tscc-1-53 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026316400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:49 tscc-1-45 kernel: Lustre: 2750:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385082/real 1408385082]  req@ffff880ee6212400 x1476537015504940/t0(0) o400->dolphin-MDT0000-mdc-ffff882028c86c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408385089 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:49 tscc-1-45 kernel: Lustre: 2750:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 11:04:49 tscc-1-45 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028c86c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:04:50 tscc-1-48 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385084/real 1408385084]  req@ffff8810194ea400 x1476537001877788/t0(0) o38->dolphin-MDT0000-mdc-ffff882028f17800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385090 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:50 tscc-0-10 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385084/real 1408385084]  req@ffff880824b6f400 x1476547227451628/t0(0) o38->dolphin-MDT0000-mdc-ffff8810322ef000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385090 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:50 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385084/real 1408385084]  req@ffff881959c63000 x1476519511704828/t0(0) o38->dolphin-MDT0000-mdc-ffff8810266d7400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385090 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:50 tscc-3-43 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385084/real 1408385084]  req@ffff8805fe1a7000 x1476519948879852/t0(0) o38->dolphin-MDT0000-mdc-ffff88101eab6000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385090 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:50 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385084/real 1408385084]  req@ffff88028853a400 x1476547204772740/t0(0) o38->dolphin-MDT0000-mdc-ffff880831f26c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385090 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:50 tscc-3-48 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385084/real 1408385084]  req@ffff881c88a94c00 x1476287317258084/t0(0) o38->dolphin-MDT0000-mdc-ffff8810266f7400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385090 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:50 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385084/real 1408385084]  req@ffff880eb04ab000 x1476519881787628/t0(0) o38->dolphin-MDT0000-mdc-ffff881025aa6800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385090 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:50 tscc-0-2 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385084/real 1408385084]  req@ffff880816376c00 x1476547435081524/t0(0) o38->dolphin-MDT0000-mdc-ffff88083175c400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385090 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:52 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385086/real 1408385086]  req@ffff88101ad52c00 x1476536956399284/t0(0) o38->dolphin-MDT0000-mdc-ffff882028ff0000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385092 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:54 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385088/real 1408385088]  req@ffff88081b23cc00 x1476547271400292/t0(0) o38->dolphin-MDT0000-mdc-ffff880833252000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385094 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:54 tscc-1-35 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385088/real 1408385088]  req@ffff8810244b6800 x1476285713815112/t0(0) o38->dolphin-MDT0000-mdc-ffff881024c5a000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385094 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:54 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385088/real 1408385088]  req@ffff88100918c800 x1476537036491432/t0(0) o38->dolphin-MDT0000-mdc-ffff882026316400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385094 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:55 tscc-1-45 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385089/real 1408385089]  req@ffff8810282d2c00 x1476537015505200/t0(0) o38->dolphin-MDT0000-mdc-ffff882028c86c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385095 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:56 tscc-1-14 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385090/real 1408385090]  req@ffff88400fe5e800 x1476519630560620/t0(0) o38->dolphin-MDT0000-mdc-ffff884010213000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385096 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:56 tscc-1-14 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:57 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385091/real 1408385091]  req@ffff8818f4dff000 x1476519531663068/t0(0) o38->dolphin-MDT0000-mdc-ffff882026059c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385097 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:57 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385091/real 1408385091]  req@ffff8808214f5000 x1476547206151952/t0(0) o38->dolphin-MDT0000-mdc-ffff8808290f1c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385097 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:57 tscc-0-62 kernel: Lustre: 2458:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385091/real 1408385091]  req@ffff881031387800 x1475809553555908/t0(0) o38->dolphin-MDT0000-mdc-ffff881033ef6400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385097 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:57 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385091/real 1408385091]  req@ffff880404cf2000 x1476519742437128/t0(0) o38->dolphin-MDT0000-mdc-ffff882026d94400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385097 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:57 tscc-0-20 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385091/real 1408385091]  req@ffff88081845d800 x1476638723838448/t0(0) o38->dolphin-MDT0000-mdc-ffff8810332bc400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385097 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:57 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385091/real 1408385091]  req@ffff880d43d8cc00 x1475562021670072/t0(0) o38->dolphin-MDT0000-mdc-ffff881032828c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385097 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:57 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385091/real 1408385091]  req@ffff880477571800 x1476536280043472/t0(0) o38->dolphin-MDT0000-mdc-ffff880876dd2c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385097 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:57 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385091/real 1408385091]  req@ffff8808303ec800 x1476536929558288/t0(0) o38->dolphin-MDT0000-mdc-ffff88103115e400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385097 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:57 tscc-gpu-5-4 kernel: Lustre: 2620:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385091/real 1408385091]  req@ffff8808548f0c00 x1476612337751888/t0(0) o38->dolphin-MDT0000-mdc-ffff880877501800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385097 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:57 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385091/real 1408385091]  req@ffff880b63b5a400 x1476519055828584/t0(0) o38->dolphin-MDT0000-mdc-ffff881065172800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385097 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:58 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385092/real 1408385092]  req@ffff880fa0ed2000 x1476290267142056/t0(0) o38->dolphin-MDT0000-mdc-ffff881067a76000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385098 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:58 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385092/real 1408385092]  req@ffff880725d2ec00 x1476612877232132/t0(0) o38->dolphin-MDT0000-mdc-ffff8808300d3800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385098 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:04:58 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:04:59 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385090/real 1408385090]  req@ffff88037e480c00 x1476710234967808/t0(0) o38->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385099 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:01 tscc-2-10 kernel: Lustre: 2962:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385095/real 1408385095]  req@ffff880ffb1bb400 x1476519079943796/t0(0) o38->dolphin-MDT0000-mdc-ffff881066654400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385101 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:08 tscc-1-40 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385102/real 1408385102]  req@ffff880ee5917000 x1476537020764516/t0(0) o38->dolphin-MDT0000-mdc-ffff881025cb7400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385108 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:10 tscc-gpu-3-1 kernel: Lustre: 2906:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385104/real 1408385104]  req@ffff880878a7d000 x1476798077475664/t0(0) o38->dolphin-MDT0000-mdc-ffff880878f15400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385110 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:11 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385105/real 1408385105]  req@ffff88077fa2a400 x1476519829338040/t0(0) o38->dolphin-MDT0000-mdc-ffff8808317cc800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385111 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:11 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385105/real 1408385105]  req@ffff881dde29dc00 x1476439898250408/t0(0) o38->dolphin-MDT0000-mdc-ffff8810258fb000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385111 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:12 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385106/real 1408385106]  req@ffff880402e65800 x1476612857973056/t0(0) o38->dolphin-MDT0000-mdc-ffff880830a6b000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385112 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:12 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385106/real 1408385106]  req@ffff881011822800 x1476547206491020/t0(0) o38->dolphin-MDT0000-mdc-ffff880831447800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385112 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:15 tscc-1-48 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385109/real 1408385109]  req@ffff881026bbf800 x1476537001877796/t0(0) o38->dolphin-MDT0000-mdc-ffff882028f17800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:15 tscc-0-10 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385109/real 1408385109]  req@ffff880826789800 x1476547227451636/t0(0) o38->dolphin-MDT0000-mdc-ffff8810322ef000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:15 tscc-login1 kernel: Lustre: Evicted from dolphin-MDT0000_UUID (at 172.25.32.25@tcp) after server handle changed from 0x30845f5537760407 to 0x30845f5537cf0aa2
Aug 18 11:05:15 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:05:15 tscc-login1 kernel: LustreError: 57369:0:(file.c:171:ll_close_inode_openhandle()) inode 144115538216353952 mdc close failed: rc = -5
Aug 18 11:05:15 tscc-login1 kernel: LustreError: 55183:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:05:15 tscc-login1 kernel: LustreError: 55183:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 1 previous similar message
Aug 18 11:05:15 tscc-login1 kernel: LustreError: 55183:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005186:0x2a:0x0] error -5.
Aug 18 11:05:15 tscc-login1 kernel: LustreError: 57369:0:(file.c:171:ll_close_inode_openhandle()) Skipped 15 previous similar messages
Aug 18 11:05:15 tscc-login1 kernel: LustreError: 57465:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID [0x20000401a:0x1823a:0x0] error: rc = -108
Aug 18 11:05:15 tscc-login1 kernel: LustreError: 57465:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 1 previous similar message
Aug 18 11:05:15 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:15 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385109/real 1408385109]  req@ffff882025ccd400 x1476519511704836/t0(0) o38->dolphin-MDT0000-mdc-ffff8810266d7400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:15 tscc-3-43 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385109/real 1408385109]  req@ffff88100495b000 x1476519948879860/t0(0) o38->dolphin-MDT0000-mdc-ffff88101eab6000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:15 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385109/real 1408385109]  req@ffff880832b32800 x1476547204772748/t0(0) o38->dolphin-MDT0000-mdc-ffff880831f26c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:15 tscc-1-14 kernel: Lustre: dolphin-MDT0000-mdc-ffff884010213000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:15 tscc-3-48 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385109/real 1408385109]  req@ffff881bec497400 x1476287317258092/t0(0) o38->dolphin-MDT0000-mdc-ffff8810266f7400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:15 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385109/real 1408385109]  req@ffff881008ab8400 x1476519881787636/t0(0) o38->dolphin-MDT0000-mdc-ffff881025aa6800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:15 tscc-0-2 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385109/real 1408385109]  req@ffff8808162a8c00 x1476547435081532/t0(0) o38->dolphin-MDT0000-mdc-ffff88083175c400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:16 tscc-1-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026059c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:16 tscc-1-1 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:16 tscc-2-39 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808290f1c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:16 tscc-2-39 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:16 tscc-0-62 kernel: Lustre: dolphin-MDT0000-mdc-ffff881033ef6400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:16 tscc-0-62 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:16 tscc-1-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026d94400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:16 tscc-1-11 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:16 tscc-0-20 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810332bc400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:16 tscc-0-20 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:16 tscc-0-57 kernel: Lustre: dolphin-MDT0000-mdc-ffff881032828c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:16 tscc-0-57 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:16 tscc-gpu-7-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876dd2c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:16 tscc-gpu-7-7 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:16 tscc-1-61 kernel: Lustre: dolphin-MDT0000-mdc-ffff88103115e400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:16 tscc-1-61 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:16 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:16 tscc-gpu-5-4 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:16 tscc-2-13 kernel: Lustre: dolphin-MDT0000-mdc-ffff881065172800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:16 tscc-2-13 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:17 tscc-2-52 kernel: Lustre: dolphin-MDT0000-mdc-ffff881067a76000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:17 tscc-2-52 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:17 tscc-0-15 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808300d3800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:17 tscc-0-15 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:17 tscc-3-40 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385111/real 1408385111]  req@ffff8810236cd800 x1476520324377008/t0(0) o38->dolphin-MDT0000-mdc-ffff881025a8d400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385117 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:17 tscc-3-40 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:05:17 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385111/real 1408385111]  req@ffff88101b23a800 x1476536956399292/t0(0) o38->dolphin-MDT0000-mdc-ffff882028ff0000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385117 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:19 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385113/real 1408385113]  req@ffff88081ca91c00 x1476547271400300/t0(0) o38->dolphin-MDT0000-mdc-ffff880833252000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385119 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:19 tscc-1-35 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385113/real 1408385113]  req@ffff880f9be7fc00 x1476285713815120/t0(0) o38->dolphin-MDT0000-mdc-ffff881024c5a000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385119 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:19 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385113/real 1408385113]  req@ffff880f83acb800 x1476537036491440/t0(0) o38->dolphin-MDT0000-mdc-ffff882026316400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385119 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:20 tscc-1-45 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385114/real 1408385114]  req@ffff881013acb400 x1476537015505208/t0(0) o38->dolphin-MDT0000-mdc-ffff882028c86c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385120 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:05:20 tscc-2-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff881066654400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:20 tscc-2-10 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:27 tscc-1-40 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025cb7400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:29 tscc-gpu-3-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880878f15400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:30 tscc-3-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808317cc800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:30 tscc-3-56 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:30 tscc-1-32 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810258fb000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:30 tscc-1-32 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:31 tscc-0-31 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830a6b000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:31 tscc-0-31 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:31 tscc-0-28 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831447800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:31 tscc-0-28 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:34 tscc-1-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028f17800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:34 tscc-1-48 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:34 tscc-0-10 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810322ef000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:34 tscc-0-10 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:34 tscc-1-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810266d7400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:34 tscc-1-6 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:34 tscc-3-43 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101eab6000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:34 tscc-3-43 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:34 tscc-0-36 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831f26c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:34 tscc-0-36 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:34 tscc-3-48 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810266f7400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:34 tscc-3-48 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:34 tscc-3-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025aa6800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:34 tscc-0-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff88083175c400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:34 tscc-0-2 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:36 tscc-3-40 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025a8d400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:36 tscc-1-56 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028ff0000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:36 tscc-1-56 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:38 tscc-0-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880833252000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:38 tscc-0-7 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:38 tscc-1-35 kernel: Lustre: dolphin-MDT0000-mdc-ffff881024c5a000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:38 tscc-1-35 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:38 tscc-1-53 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026316400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:38 tscc-1-53 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:05:39 tscc-1-45 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028c86c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:05:39 tscc-1-45 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:07:26 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:07:41 tscc-1-1 kernel: Lustre: 51267:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385249/real 1408385249]  req@ffff8819be49dc00 x1476519531664912/t0(0) o35->dolphin-MDT0000-mdc-ffff882026059c00@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408385261 ref 2 fl Rpc:IX/0/ffffffff rc 0/-1
Aug 18 11:07:41 tscc-1-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026059c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:07:41 tscc-1-1 kernel: LustreError: 51267:0:(file.c:171:ll_close_inode_openhandle()) inode 144115537545265237 mdc close failed: rc = -4
Aug 18 11:07:41 tscc-1-1 kernel: LustreError: 51267:0:(file.c:171:ll_close_inode_openhandle()) Skipped 11 previous similar messages
Aug 18 11:07:42 tscc-1-1 kernel: LustreError: 51267:0:(file.c:171:ll_close_inode_openhandle()) inode 144115537545265238 mdc close failed: rc = -4
Aug 18 11:07:44 tscc-1-1 kernel: LustreError: 51267:0:(file.c:171:ll_close_inode_openhandle()) inode 144115537545265240 mdc close failed: rc = -4
Aug 18 11:07:44 tscc-1-1 kernel: LustreError: 51267:0:(file.c:171:ll_close_inode_openhandle()) Skipped 1 previous similar message
Aug 18 11:07:47 tscc-1-1 kernel: LustreError: 51267:0:(file.c:171:ll_close_inode_openhandle()) inode 144115537545265243 mdc close failed: rc = -4
Aug 18 11:07:47 tscc-1-1 kernel: LustreError: 51267:0:(file.c:171:ll_close_inode_openhandle()) Skipped 2 previous similar messages
Aug 18 11:07:50 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 11:07:50 tscc-1-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026059c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:07:50 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:07:50 tscc-login1 kernel: LustreError: 56313:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:07:50 tscc-login1 kernel: LustreError: 55183:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005187:0x83:0x0] error -5.
Aug 18 11:07:50 tscc-login1 kernel: LustreError: 55183:0:(vvp_io.c:1229:vvp_io_init()) Skipped 120 previous similar messages
Aug 18 11:07:50 tscc-login1 kernel: LustreError: 56313:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 133 previous similar messages
Aug 18 11:07:50 tscc-login1 kernel: LustreError: 59348:0:(file.c:171:ll_close_inode_openhandle()) inode 144115463390069862 mdc close failed: rc = -108
Aug 18 11:07:50 tscc-login1 kernel: LustreError: 59348:0:(file.c:171:ll_close_inode_openhandle()) Skipped 36 previous similar messages
Aug 18 11:07:50 tscc-login1 kernel: LustreError: 4049:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff8810311a7800: Can't allocate new meta-sequence,rc -108
Aug 18 11:07:50 tscc-login1 kernel: LustreError: 4049:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff8810311a7800: Can't allocate new sequence, rc -108
Aug 18 11:07:50 tscc-login1 kernel: LustreError: 4049:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 11:07:50 tscc-login1 kernel: LustreError: 59349:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID [0x250029c7:0x87004be5:0x0] error: rc = -108
Aug 18 11:07:50 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:08:49 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 11:08:49 tscc-login1 kernel: LustreError: Skipped 3 previous similar messages
Aug 18 11:08:49 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:08:49 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:08:49 tscc-login1 kernel: LustreError: 56313:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:08:49 tscc-login1 kernel: LustreError: 55183:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005185:0x4f:0x0] error -5.
Aug 18 11:08:49 tscc-login1 kernel: LustreError: 55183:0:(vvp_io.c:1229:vvp_io_init()) Skipped 5 previous similar messages
Aug 18 11:08:49 tscc-login1 kernel: LustreError: 56313:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 30 previous similar messages
Aug 18 11:08:49 tscc-login1 kernel: LustreError: 59642:0:(file.c:171:ll_close_inode_openhandle()) inode 144115524375200341 mdc close failed: rc = -108
Aug 18 11:08:49 tscc-login1 kernel: LustreError: 59642:0:(file.c:171:ll_close_inode_openhandle()) Skipped 26 previous similar messages
Aug 18 11:08:49 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:08:49 tscc-login1 kernel: LustreError: 56313:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID [0x14e00001:0xa0252bbc:0x0] error: rc = -108
Aug 18 11:08:49 tscc-login1 kernel: LustreError: 56313:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 3 previous similar messages
Aug 18 11:09:50 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 11:09:50 tscc-login1 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 11:09:50 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:09:50 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:09:50 tscc-login1 kernel: LustreError: 60328:0:(file.c:171:ll_close_inode_openhandle()) inode 144115535431336302 mdc close failed: rc = -5
Aug 18 11:09:50 tscc-login1 kernel: LustreError: 55183:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:09:50 tscc-login1 kernel: LustreError: 55183:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 5 previous similar messages
Aug 18 11:09:50 tscc-login1 kernel: LustreError: 55183:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005189:0x8:0x0] error -5.
Aug 18 11:09:50 tscc-login1 kernel: LustreError: 55183:0:(vvp_io.c:1229:vvp_io_init()) Skipped 2 previous similar messages
Aug 18 11:09:50 tscc-login1 kernel: LustreError: 60328:0:(file.c:171:ll_close_inode_openhandle()) Skipped 2 previous similar messages
Aug 18 11:09:50 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:13:25 tscc-gpu-3-3 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 11:13:30 tscc-gpu-3-3 kernel: Lustre: 2893:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385605/real 1408385605]  req@ffff8810726f2c00 x1476799344148484/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408385610 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:13:31 tscc-gpu-3-3 kernel: LustreError: 2829:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff8810726f2800 x1476799344148488/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 11:13:40 tscc-gpu-3-3 kernel: LustreError: 2923:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880870022c00 x1476799344148496/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 11:13:55 tscc-gpu-3-3 kernel: Lustre: 2893:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408385630/real 1408385630]  req@ffff88086fa05800 x1476799344148580/t0(0) o38->dolphin-MDT0000-mdc-ffff8810711e2800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408385635 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:14:15 tscc-gpu-3-3 kernel: Lustre: Layout lock feature supported.
Aug 18 11:14:15 tscc-gpu-3-3 kernel: Lustre: Mounted dolphin-client
Aug 18 11:26:16 tscc-0-0 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386365/real 1408386365]  req@ffff88101d791c00 x1476519380600076/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:16 tscc-0-0 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:16 tscc-0-39 kernel: Lustre: 2225:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386365/real 1408386365]  req@ffff8803ee21dc00 x1475608471411724/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:16 tscc-0-39 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:16 tscc-3-46 kernel: Lustre: 56553:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386365/real 1408386365]  req@ffff8820280fa800 x1476519914723192/t0(0) o35->dolphin-MDT0000-mdc-ffff882028f52400@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408386376 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 11:26:16 tscc-3-46 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028f52400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:26:16 tscc-0-21 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386365/real 1408386365]  req@ffff881033ef7c00 x1476699991531600/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:16 tscc-0-21 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:17 tscc-gpu-3-4 kernel: Lustre: 2565:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff881059693c00 x1476571902028532/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:17 tscc-gpu-3-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:17 tscc-pdafm-7-0 kernel: Lustre: 3168:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff887ffff1f400 x1476534244791988/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:17 tscc-pdafm-7-0 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:17 tscc-gpu-5-2 kernel: Lustre: 2629:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff880854a52400 x1476558499480640/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:17 tscc-gpu-5-2 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:17 tscc-0-18 kernel: Lustre: 2763:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff88081f1d5000 x1476547249474664/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:17 tscc-0-18 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:17 tscc-3-46 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff880e088a2400 x1476519914723204/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:17 tscc-3-46 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:17 tscc-0-13 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff8810328d5400 x1476634825390348/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:17 tscc-0-13 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:17 tscc-3-49 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff880d8451f000 x1476519923735900/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:17 tscc-3-49 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:17 tscc-2-37 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff880708c52800 x1476570224059916/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:17 tscc-2-37 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:18 tscc-0-5 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386367/real 1408386367]  req@ffff880814c4fc00 x1476638712318688/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386378 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:18 tscc-0-5 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:19 tscc-1-12 kernel: Lustre: 2768:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386368/real 1408386368]  req@ffff881657d4e800 x1476547036055872/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386379 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:19 tscc-1-12 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:20 tscc-0-60 kernel: Lustre: 2822:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386368/real 1408386368]  req@ffff881031f2b800 x1476700762674444/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386380 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:20 tscc-0-60 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:20 tscc-gpu-3-2 kernel: Lustre: 2906:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386368/real 1408386368]  req@ffff88087a644c00 x1476798148791496/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386380 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:20 tscc-gpu-3-2 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:20 tscc-1-33 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386368/real 1408386368]  req@ffff881026c30800 x1476536972523776/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386380 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:20 tscc-1-33 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:21 tscc-3-41 kernel: Lustre: 12725:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff88099299a000 x1476519856701792/t0(0) o35->dolphin-MDT0000-mdc-ffff882026020800@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408386381 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 11:26:21 tscc-3-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026020800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:26:21 tscc-1-46 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386369/real 1408386369]  req@ffff880f71edbc00 x1476536973594736/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386381 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:21 tscc-1-46 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:21 tscc-1-41 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386370/real 1408386370]  req@ffff880d0bf95c00 x1476537045916384/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386381 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:21 tscc-1-41 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:21 tscc-1-59 kernel: Lustre: 2740:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386369/real 1408386369]  req@ffff88081d8ebc00 x1476536863839436/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386381 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:21 tscc-1-59 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:22 tscc-0-0 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386376/real 1408386376]  req@ffff880a8d6f5c00 x1476519380600340/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386382 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:22 tscc-0-39 kernel: Lustre: 2216:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386376/real 1408386376]  req@ffff8804d4374400 x1475608471411988/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386382 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:22 tscc-0-21 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386376/real 1408386376]  req@ffff8810318bcc00 x1476699991531864/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386382 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:23 tscc-gpu-3-4 kernel: Lustre: 2555:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386377/real 1408386377]  req@ffff881055441000 x1476571902028784/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386383 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:23 tscc-pdafm-7-0 kernel: Lustre: 3148:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386377/real 1408386377]  req@ffff88800063bc00 x1476534244792252/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386383 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:23 tscc-gpu-5-2 kernel: Lustre: 2625:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386377/real 1408386377]  req@ffff880828ae7400 x1476558499480840/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386383 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:23 tscc-0-18 kernel: Lustre: 2753:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386377/real 1408386377]  req@ffff88081f045000 x1476547249474928/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386383 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:23 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386377/real 1408386377]  req@ffff880d27348800 x1476519914723328/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386383 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:23 tscc-0-13 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386377/real 1408386377]  req@ffff881033227800 x1476634825390612/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386383 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:23 tscc-3-49 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386377/real 1408386377]  req@ffff880c73245c00 x1476519923736164/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386383 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:23 tscc-2-37 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386377/real 1408386377]  req@ffff88056fb2fc00 x1476570224060180/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386383 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:24 tscc-0-5 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386378/real 1408386378]  req@ffff880812a54c00 x1476638712318952/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386384 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:25 tscc-1-12 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386379/real 1408386379]  req@ffff88085f85b400 x1476547036056136/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386385 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:25 tscc-1-38 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386372/real 1408386372]  req@ffff880c2e7d3000 x1476575145559412/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386385 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:25 tscc-1-38 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:25 tscc-2-50 kernel: Lustre: 2961:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386372/real 1408386372]  req@ffff881c15467000 x1476518982737180/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386385 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:25 tscc-2-50 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:25 tscc-1-51 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386372/real 1408386372]  req@ffff881c29796000 x1476542205929216/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386385 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:25 tscc-1-51 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:25 tscc-gpu-6-1 kernel: Lustre: 2533:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386372/real 1408386372]  req@ffff88081074b400 x1476558706386472/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386385 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:25 tscc-gpu-6-1 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:25 tscc-0-26 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386373/real 1408386373]  req@ffff880f3b367400 x1476612864111656/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386385 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:25 tscc-0-26 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:25 tscc-gpu-6-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476c19c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:26:25 tscc-gpu-6-6 kernel: Lustre: 2528:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386373/real 1408386373]  req@ffff88086de38400 x1476558757918376/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386385 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:25 tscc-gpu-6-6 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:25 tscc-gpu-5-7 kernel: Lustre: 2563:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386373/real 1408386373]  req@ffff88046eb16800 x1476612406249200/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386385 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:25 tscc-gpu-5-7 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:25 tscc-0-29 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386373/real 1408386373]  req@ffff880831bd6800 x1476612872576224/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386385 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:25 tscc-0-29 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:25 tscc-0-8 kernel: Lustre: 2763:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386373/real 1408386373]  req@ffff880814607c00 x1476547167665688/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386385 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:25 tscc-0-8 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:26 tscc-0-34 kernel: Lustre: 2757:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386372/real 1408386372]  req@ffff88082e99a800 x1476614108114556/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386386 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:26 tscc-0-34 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:26 tscc-1-43 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386375/real 1408386375]  req@ffff880d79283c00 x1476570186886588/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386386 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:26 tscc-1-43 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:27 tscc-0-60 kernel: Lustre: 2813:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386380/real 1408386380]  req@ffff880ff47d3c00 x1476700762674708/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386387 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:27 tscc-1-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386381/real 1408386381]  req@ffff880ff598b800 x1476537045916648/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386387 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:27 tscc-1-33 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386380/real 1408386380]  req@ffff880ffa496000 x1476536972524040/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386387 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:27 tscc-gpu-3-2 kernel: Lustre: 2905:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386380/real 1408386380]  req@ffff880878013400 x1476798148791760/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386387 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:28 tscc-1-46 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386381/real 1408386381]  req@ffff881019bc2000 x1476536973595000/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386388 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:28 tscc-1-59 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386381/real 1408386381]  req@ffff88060f83cc00 x1476536863839700/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386388 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:30 tscc-1-54 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386376/real 1408386376]  req@ffff880f163dac00 x1476536988250432/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386390 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:30 tscc-1-54 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:32 tscc-1-43 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386386/real 1408386386]  req@ffff880dfb81a800 x1476570186886852/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386392 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:32 tscc-0-26 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386385/real 1408386385]  req@ffff880f3d594c00 x1476612864111920/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386392 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:32 tscc-gpu-6-6 kernel: Lustre: 2524:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386385/real 1408386385]  req@ffff880855c29800 x1476558757918640/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386392 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:32 tscc-gpu-5-7 kernel: Lustre: 2551:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386385/real 1408386385]  req@ffff880458e6cc00 x1476612406249340/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386392 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:32 tscc-0-29 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386385/real 1408386385]  req@ffff8804df082c00 x1476612872576488/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386392 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:32 tscc-0-8 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386385/real 1408386385]  req@ffff88081527cc00 x1476547167665952/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386392 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:33 tscc-1-38 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386385/real 1408386385]  req@ffff881028627800 x1476575145559676/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386393 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:33 tscc-2-50 kernel: Lustre: 2956:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386385/real 1408386385]  req@ffff881dc6fba800 x1476518982737444/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386393 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:33 tscc-1-51 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386385/real 1408386385]  req@ffff881cab791c00 x1476542205929480/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386393 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:33 tscc-3-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386381/real 1408386381]  req@ffff880e8484bc00 x1476519856702008/t0(0) o38->dolphin-MDT0000-mdc-ffff882026020800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386393 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:33 tscc-gpu-6-1 kernel: Lustre: 2523:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386385/real 1408386385]  req@ffff88086713b400 x1476558706386728/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386393 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:33 tscc-gpu-6-1 kernel: Lustre: 2523:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:26:34 tscc-1-9 kernel: Lustre: 2739:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386382/real 1408386382]  req@ffff8809d66c7800 x1476519677615816/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386394 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:34 tscc-1-9 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:34 tscc-2-11 kernel: Lustre: 2967:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386379/real 1408386379]  req@ffff88105b8bd800 x1476519079108184/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386394 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:34 tscc-2-11 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:34 tscc-3-41 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386377/real 1408386377]  req@ffff881004b08400 x1476519856701816/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386394 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:34 tscc-3-41 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:35 tscc-0-34 kernel: Lustre: 2756:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386386/real 1408386386]  req@ffff8807735aa000 x1476614108114820/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386395 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:36 tscc-1-4 kernel: Lustre: 2743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386385/real 1408386385]  req@ffff880b5d672000 x1475918802709160/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386396 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:36 tscc-1-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:37 tscc-0-63 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386386/real 1408386386]  req@ffff8810323adc00 x1476798012478200/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386397 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:37 tscc-0-63 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:38 tscc-gpu-6-1 kernel: Lustre: 2523:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386385/real 1408386385]  req@ffff88086713b000 x1476558706386736/t0(0) o38->dolphin-MDT0000-mdc-ffff880476c19c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386398 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:39 tscc-1-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386390/real 1408386390]  req@ffff880d0b33b000 x1476536988250696/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386399 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:40 tscc-3-54 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386389/real 1408386389]  req@ffff880ea1e65800 x1476290785566560/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386400 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:40 tscc-3-54 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:41 tscc-2-55 kernel: Lustre: 2969:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386390/real 1408386390]  req@ffff881fa46a6000 x1476519002406432/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386401 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:41 tscc-2-55 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:26:41 tscc-1-9 kernel: Lustre: 2738:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386394/real 1408386394]  req@ffff8811b2826000 x1476519677616080/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386401 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:42 tscc-1-4 kernel: Lustre: 2732:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386396/real 1408386396]  req@ffff880b5d672000 x1475918802709408/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386402 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:43 tscc-0-63 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386397/real 1408386397]  req@ffff880fe9e17400 x1476798012478464/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386403 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:44 tscc-2-11 kernel: Lustre: 2959:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386394/real 1408386394]  req@ffff8801593ed400 x1476519079108448/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386404 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:46 tscc-3-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386400/real 1408386400]  req@ffff880f062a2000 x1476290785566824/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386406 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:46 tscc-3-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386394/real 1408386394]  req@ffff880904633400 x1476519856702044/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386406 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:47 tscc-2-55 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386401/real 1408386401]  req@ffff881f2a9fd800 x1476519002406696/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386407 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:47 tscc-0-0 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386401/real 1408386401]  req@ffff8810310a0000 x1476519380600344/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386407 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:47 tscc-0-39 kernel: Lustre: 2216:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386401/real 1408386401]  req@ffff8805952ddc00 x1475608471411992/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386407 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:47 tscc-0-21 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386401/real 1408386401]  req@ffff881033ef7400 x1476699991531868/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386407 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:48 tscc-gpu-3-4 kernel: Lustre: 2555:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386402/real 1408386402]  req@ffff88105e6d3800 x1476571902028800/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386408 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:48 tscc-pdafm-7-0 kernel: Lustre: 3148:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386402/real 1408386402]  req@ffff887ffff1f400 x1476534244792256/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386408 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:48 tscc-gpu-5-2 kernel: Lustre: 2625:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386402/real 1408386402]  req@ffff880848e37400 x1476558499481076/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386408 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:48 tscc-0-18 kernel: Lustre: 2753:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386402/real 1408386402]  req@ffff880820b02400 x1476547249474932/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386408 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:48 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386402/real 1408386402]  req@ffff88101b6c0c00 x1476519914723364/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386408 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:48 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:26:48 tscc-0-13 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386402/real 1408386402]  req@ffff880ac8359000 x1476634825390616/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386408 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:48 tscc-3-49 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386402/real 1408386402]  req@ffff880d8451f400 x1476519923736168/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386408 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:48 tscc-2-37 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386402/real 1408386402]  req@ffff880188369000 x1476570224060184/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386408 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:49 tscc-0-5 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386403/real 1408386403]  req@ffff880814c4f000 x1476638712318956/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386409 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:50 tscc-1-12 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386404/real 1408386404]  req@ffff881794990400 x1476547036056140/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386410 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:52 tscc-0-60 kernel: Lustre: 2813:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386405/real 1408386405]  req@ffff881030ea3c00 x1476700762674712/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386412 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:52 tscc-1-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386406/real 1408386406]  req@ffff880ff598b000 x1476537045916652/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386412 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:52 tscc-1-33 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386405/real 1408386405]  req@ffff880ca8254400 x1476536972524044/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386412 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:52 tscc-gpu-3-2 kernel: Lustre: 2905:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386405/real 1408386405]  req@ffff88087a644c00 x1476798148791764/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386412 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:53 tscc-1-46 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386406/real 1408386406]  req@ffff880f71edbc00 x1476536973595004/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386413 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:53 tscc-1-59 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386406/real 1408386406]  req@ffff88081d8eb800 x1476536863839704/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386413 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:57 tscc-1-43 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386411/real 1408386411]  req@ffff880d79283800 x1476570186886856/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386417 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:57 tscc-0-26 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386410/real 1408386410]  req@ffff880f58055800 x1476612864111924/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386417 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:57 tscc-gpu-6-6 kernel: Lustre: 2524:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386410/real 1408386410]  req@ffff88086a7cd000 x1476558757918644/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386417 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:57 tscc-gpu-5-7 kernel: Lustre: 2551:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386410/real 1408386410]  req@ffff8804525c6c00 x1476612406249484/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386417 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:57 tscc-0-29 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386410/real 1408386410]  req@ffff8803b2509000 x1476612872576492/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386417 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:57 tscc-0-8 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386410/real 1408386410]  req@ffff880814f7fc00 x1476547167665956/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386417 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:58 tscc-1-38 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386410/real 1408386410]  req@ffff880c172ad000 x1476575145559680/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386418 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:58 tscc-2-50 kernel: Lustre: 2956:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386410/real 1408386410]  req@ffff881f71f4dc00 x1476518982737448/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386418 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:58 tscc-1-51 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386410/real 1408386410]  req@ffff882025a86800 x1476542205929484/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386418 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:26:58 tscc-gpu-6-1 kernel: Lustre: 2523:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386410/real 1408386410]  req@ffff88087728fc00 x1476558706386740/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386418 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:00 tscc-0-34 kernel: Lustre: 2756:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386411/real 1408386411]  req@ffff88082e99a400 x1476614108114824/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386420 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:02 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386402/real 1408386402]  req@ffff88101b6c0000 x1476519914723368/t0(0) o38->dolphin-MDT0000-mdc-ffff882028f52400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386422 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:03 tscc-0-0 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386365/real 1408386365]  req@ffff8810310a0c00 x1476519380600080/t0(0) o400->dolphin-MDT0000-mdc-ffff8810326bb000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386423 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:03 tscc-0-0 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810326bb000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:03 tscc-gpu-6-1 kernel: Lustre: 2523:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386410/real 1408386410]  req@ffff88087728f400 x1476558706386744/t0(0) o38->dolphin-MDT0000-mdc-ffff880476c19c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386423 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:03 tscc-gpu-6-1 kernel: Lustre: 2523:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:27:03 tscc-0-39 kernel: Lustre: 2226:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386365/real 1408386365]  req@ffff8807c9519000 x1475608471411728/t0(0) o400->dolphin-MDT0000-mdc-ffff88081c7a2000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386423 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:03 tscc-0-39 kernel: Lustre: dolphin-MDT0000-mdc-ffff88081c7a2000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:03 tscc-gpu-5-7 kernel: Lustre: 2552:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386376/real 1408386376]  req@ffff880869eed800 x1476612406249336/t0(0) o103->dolphin-MDT0000-mdc-ffff880876ea6c00@172.25.32.25@tcp:17/18 lens 328/224 e 0 to 1 dl 1408386423 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 11:27:03 tscc-gpu-5-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876ea6c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:03 tscc-0-21 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386365/real 1408386365]  req@ffff881033ef7000 x1476699991531604/t0(0) o400->dolphin-MDT0000-mdc-ffff881032040400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386423 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:03 tscc-0-21 kernel: Lustre: dolphin-MDT0000-mdc-ffff881032040400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:04 tscc-gpu-3-4 kernel: Lustre: 2566:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff881059693800 x1476571902028536/t0(0) o400->dolphin-MDT0000-mdc-ffff88087229d800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386424 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:04 tscc-gpu-3-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff88087229d800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:04 tscc-pdafm-7-0 kernel: Lustre: 3170:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff88800056c800 x1476534244791992/t0(0) o400->dolphin-MDT0000-mdc-ffff885010079800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386424 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:04 tscc-pdafm-7-0 kernel: Lustre: dolphin-MDT0000-mdc-ffff885010079800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:04 tscc-0-18 kernel: Lustre: 2764:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff88081f1d5800 x1476547249474668/t0(0) o400->dolphin-MDT0000-mdc-ffff8810343ce400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386424 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:04 tscc-0-18 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810343ce400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:04 tscc-0-13 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff8809e5cb9c00 x1476634825390352/t0(0) o400->dolphin-MDT0000-mdc-ffff880830ad3800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386424 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:04 tscc-0-13 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830ad3800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:04 tscc-1-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386415/real 1408386415]  req@ffff881014a13c00 x1476536988250700/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386424 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:04 tscc-3-49 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff880d8451fc00 x1476519923735904/t0(0) o400->dolphin-MDT0000-mdc-ffff882026177c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386424 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:04 tscc-3-49 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026177c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:04 tscc-2-37 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff88081d770000 x1476570224059920/t0(0) o400->dolphin-MDT0000-mdc-ffff881033c5a800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386424 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:04 tscc-2-37 kernel: Lustre: dolphin-MDT0000-mdc-ffff881033c5a800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:05 tscc-0-5 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386367/real 1408386367]  req@ffff880814c4f800 x1476638712318692/t0(0) o400->dolphin-MDT0000-mdc-ffff88103170a000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386425 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:05 tscc-0-5 kernel: Lustre: dolphin-MDT0000-mdc-ffff88103170a000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:06 tscc-1-12 kernel: Lustre: 2769:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386368/real 1408386368]  req@ffff881657d4e000 x1476547036055876/t0(0) o400->dolphin-MDT0000-mdc-ffff882011047000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386426 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:06 tscc-1-12 kernel: Lustre: dolphin-MDT0000-mdc-ffff882011047000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:06 tscc-1-9 kernel: Lustre: 2738:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386419/real 1408386419]  req@ffff8815ae02cc00 x1476519677616084/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386426 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:07 tscc-1-4 kernel: Lustre: 2732:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386421/real 1408386421]  req@ffff880ca8193c00 x1475918802709424/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:07 tscc-0-60 kernel: Lustre: 2823:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386368/real 1408386368]  req@ffff881031f2b400 x1476700762674448/t0(0) o400->dolphin-MDT0000-mdc-ffff881030ca2400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:07 tscc-0-60 kernel: Lustre: dolphin-MDT0000-mdc-ffff881030ca2400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:07 tscc-1-33 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386368/real 1408386368]  req@ffff88102582a000 x1476536972523780/t0(0) o400->dolphin-MDT0000-mdc-ffff8810264d1400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:07 tscc-1-33 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810264d1400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:07 tscc-gpu-3-2 kernel: Lustre: 2906:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386368/real 1408386368]  req@ffff88087a644000 x1476798148791500/t0(0) o400->dolphin-MDT0000-mdc-ffff8808767dd000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:07 tscc-gpu-3-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808767dd000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:08 tscc-0-63 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386422/real 1408386422]  req@ffff8810323ad800 x1476798012478468/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386428 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:08 tscc-gpu-5-2 kernel: Lustre: 28087:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386366/real 1408386366]  req@ffff88080c6d4000 x1476558499480636/t0(0) o101->dolphin-MDT0000-mdc-ffff88086f941c00@172.25.32.25@tcp:12/10 lens 616/1136 e 0 to 1 dl 1408386428 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1
Aug 18 11:27:08 tscc-gpu-5-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff88086f941c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:08 tscc-1-46 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386369/real 1408386369]  req@ffff880f71edb400 x1476536973594740/t0(0) o400->dolphin-MDT0000-mdc-ffff88101d826800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386428 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:08 tscc-1-46 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101d826800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:08 tscc-1-41 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386370/real 1408386370]  req@ffff880d0bf95800 x1476537045916388/t0(0) o400->dolphin-MDT0000-mdc-ffff882026991400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386428 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:08 tscc-1-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026991400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:08 tscc-1-59 kernel: Lustre: 2739:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386369/real 1408386369]  req@ffff88081d8eb400 x1476536863839440/t0(0) o400->dolphin-MDT0000-mdc-ffff880831706400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386428 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:08 tscc-1-59 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831706400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:09 tscc-2-11 kernel: Lustre: 2959:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386419/real 1408386419]  req@ffff88105b8bd400 x1476519079108452/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386429 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:09 tscc-0-0 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386423/real 1408386423]  req@ffff88101d59d000 x1476519380600608/t0(0) o38->dolphin-MDT0000-mdc-ffff8810326bb000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386429 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:09 tscc-0-39 kernel: Lustre: 2216:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386423/real 1408386423]  req@ffff8801fed82c00 x1475608471412256/t0(0) o38->dolphin-MDT0000-mdc-ffff88081c7a2000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386429 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:09 tscc-1-4 kernel: Lustre: 27101:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386371/real 1408386371]  req@ffff880c0f355800 x1475918802709144/t0(0) o41->dolphin-MDT0000-mdc-ffff88101ea3f000@172.25.32.25@tcp:12/10 lens 224/368 e 0 to 1 dl 1408386429 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 11:27:09 tscc-1-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101ea3f000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:09 tscc-0-21 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386423/real 1408386423]  req@ffff881032693000 x1476699991532132/t0(0) o38->dolphin-MDT0000-mdc-ffff881032040400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386429 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:10 tscc-gpu-5-7 kernel: Lustre: 30623:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386369/real 1408386369]  req@ffff880447a7ac00 x1476612406249196/t0(0) o101->dolphin-MDT0000-mdc-ffff880876ea6c00@172.25.32.25@tcp:12/10 lens 376/1928 e 0 to 1 dl 1408386430 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 11:27:10 tscc-gpu-3-4 kernel: Lustre: 2555:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386424/real 1408386424]  req@ffff88105784e800 x1476571902029060/t0(0) o38->dolphin-MDT0000-mdc-ffff88087229d800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386430 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:10 tscc-pdafm-7-0 kernel: Lustre: 3148:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386424/real 1408386424]  req@ffff8880116c8000 x1476534244792520/t0(0) o38->dolphin-MDT0000-mdc-ffff885010079800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386430 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:10 tscc-0-18 kernel: Lustre: 2753:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386424/real 1408386424]  req@ffff880832557400 x1476547249475196/t0(0) o38->dolphin-MDT0000-mdc-ffff8810343ce400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386430 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:10 tscc-0-13 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386424/real 1408386424]  req@ffff880ac8359000 x1476634825390880/t0(0) o38->dolphin-MDT0000-mdc-ffff880830ad3800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386430 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:10 tscc-3-49 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386424/real 1408386424]  req@ffff880d8451f400 x1476519923736432/t0(0) o38->dolphin-MDT0000-mdc-ffff882026177c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386430 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:10 tscc-2-37 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386424/real 1408386424]  req@ffff88081d770400 x1476570224060448/t0(0) o38->dolphin-MDT0000-mdc-ffff881033c5a800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386430 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:11 tscc-3-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386425/real 1408386425]  req@ffff880cd8c7a000 x1476290785566828/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386431 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:11 tscc-0-5 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386425/real 1408386425]  req@ffff880814c4fc00 x1476638712319220/t0(0) o38->dolphin-MDT0000-mdc-ffff88103170a000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386431 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:11 tscc-3-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386419/real 1408386419]  req@ffff880841063c00 x1476519856702096/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386431 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:12 tscc-1-12 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386426/real 1408386426]  req@ffff88057d5b1800 x1476547036056404/t0(0) o38->dolphin-MDT0000-mdc-ffff882011047000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386432 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:12 tscc-1-38 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386372/real 1408386372]  req@ffff880fe23a7000 x1476575145559416/t0(0) o400->dolphin-MDT0000-mdc-ffff882026301000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386432 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:12 tscc-1-38 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026301000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:12 tscc-2-55 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386426/real 1408386426]  req@ffff881fa46a6400 x1476519002406700/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386432 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:12 tscc-0-34 kernel: Lustre: 2757:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386372/real 1408386372]  req@ffff88082e99a000 x1476614108114560/t0(0) o400->dolphin-MDT0000-mdc-ffff881030f1ac00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386432 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:12 tscc-0-34 kernel: Lustre: dolphin-MDT0000-mdc-ffff881030f1ac00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:12 tscc-2-50 kernel: Lustre: 2961:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386372/real 1408386372]  req@ffff881f71f4d400 x1476518982737184/t0(0) o400->dolphin-MDT0000-mdc-ffff881065a37400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386432 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:12 tscc-2-50 kernel: Lustre: dolphin-MDT0000-mdc-ffff881065a37400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:12 tscc-1-51 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386372/real 1408386372]  req@ffff882007e70000 x1476542205929220/t0(0) o400->dolphin-MDT0000-mdc-ffff882025d49c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386432 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:12 tscc-1-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff882025d49c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:12 tscc-0-26 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386373/real 1408386373]  req@ffff880fc0989400 x1476612864111660/t0(0) o400->dolphin-MDT0000-mdc-ffff8810314d8800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386432 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:12 tscc-0-26 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810314d8800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:12 tscc-gpu-6-6 kernel: Lustre: 2528:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386373/real 1408386373]  req@ffff880877824800 x1476558757918380/t0(0) o400->dolphin-MDT0000-mdc-ffff880878d95000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386432 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:12 tscc-gpu-6-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff880878d95000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:12 tscc-0-29 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386373/real 1408386373]  req@ffff8804f5395400 x1476612872576228/t0(0) o400->dolphin-MDT0000-mdc-ffff8808308f6c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386432 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:12 tscc-0-29 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808308f6c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:12 tscc-0-8 kernel: Lustre: 2764:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386373/real 1408386373]  req@ffff880814607400 x1476547167665692/t0(0) o400->dolphin-MDT0000-mdc-ffff88082f334800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386432 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:12 tscc-0-8 kernel: Lustre: dolphin-MDT0000-mdc-ffff88082f334800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:13 tscc-1-43 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386375/real 1408386375]  req@ffff881008dd4c00 x1476570186886592/t0(0) o400->dolphin-MDT0000-mdc-ffff8810259ff400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386433 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:13 tscc-1-43 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810259ff400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:14 tscc-0-60 kernel: Lustre: 2813:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386427/real 1408386427]  req@ffff880ff462e400 x1476700762674976/t0(0) o38->dolphin-MDT0000-mdc-ffff881030ca2400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386434 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:14 tscc-gpu-6-1 kernel: Lustre: 2587:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386369/real 1408386369]  req@ffff88086352bc00 x1476558706384780/t0(0) o101->dolphin-MDT0000-mdc-ffff880476c19c00@172.25.32.53@tcp:12/10 lens 640/1136 e 0 to 1 dl 1408386434 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1
Aug 18 11:27:14 tscc-1-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386428/real 1408386428]  req@ffff880edfae9800 x1476537045916916/t0(0) o38->dolphin-MDT0000-mdc-ffff882026991400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386434 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:14 tscc-1-33 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386427/real 1408386427]  req@ffff880ce99db000 x1476536972524308/t0(0) o38->dolphin-MDT0000-mdc-ffff8810264d1400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386434 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:14 tscc-gpu-3-2 kernel: Lustre: 2905:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386427/real 1408386427]  req@ffff880868f33000 x1476798148792028/t0(0) o38->dolphin-MDT0000-mdc-ffff8808767dd000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386434 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:15 tscc-1-46 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386428/real 1408386428]  req@ffff880f73305c00 x1476536973595268/t0(0) o38->dolphin-MDT0000-mdc-ffff88101d826800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386435 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:15 tscc-1-59 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386428/real 1408386428]  req@ffff88081d8eb800 x1476536863839968/t0(0) o38->dolphin-MDT0000-mdc-ffff880831706400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386435 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:15 tscc-1-4 kernel: Lustre: 2732:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386429/real 1408386429]  req@ffff880ca8193400 x1475918802709688/t0(0) o38->dolphin-MDT0000-mdc-ffff88101ea3f000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386435 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:17 tscc-1-54 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386376/real 1408386376]  req@ffff881012418000 x1476536988250436/t0(0) o400->dolphin-MDT0000-mdc-ffff881025a03800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386437 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:17 tscc-1-54 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025a03800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:18 tscc-gpu-5-2 kernel: Lustre: 1961:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386376/real 1408386376]  req@ffff8808772c0000 x1476558499480812/t0(0) o44->dolphin-MDT0000-mdc-ffff88086f941c00@172.25.32.25@tcp:12/10 lens 448/440 e 0 to 1 dl 1408386438 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 11:27:18 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386427/real 1408386427]  req@ffff880d27348400 x1476519914723620/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386438 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:18 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:27:19 tscc-1-43 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386433/real 1408386433]  req@ffff881008dd4400 x1476570186887120/t0(0) o38->dolphin-MDT0000-mdc-ffff8810259ff400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386439 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:19 tscc-0-26 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386432/real 1408386432]  req@ffff881032502400 x1476612864112188/t0(0) o38->dolphin-MDT0000-mdc-ffff8810314d8800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386439 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:19 tscc-gpu-6-6 kernel: Lustre: 2524:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386432/real 1408386432]  req@ffff88086d086400 x1476558757918908/t0(0) o38->dolphin-MDT0000-mdc-ffff880878d95000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386439 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:19 tscc-0-29 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386432/real 1408386432]  req@ffff8804f5395c00 x1476612872576756/t0(0) o38->dolphin-MDT0000-mdc-ffff8808308f6c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386439 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:19 tscc-0-8 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386432/real 1408386432]  req@ffff880814bf9400 x1476547167666220/t0(0) o38->dolphin-MDT0000-mdc-ffff88082f334800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386439 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:20 tscc-1-38 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386432/real 1408386432]  req@ffff880cd7cbf800 x1476575145559944/t0(0) o38->dolphin-MDT0000-mdc-ffff882026301000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386440 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:20 tscc-0-34 kernel: Lustre: 2756:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386432/real 1408386432]  req@ffff880796f93000 x1476614108115088/t0(0) o38->dolphin-MDT0000-mdc-ffff881030f1ac00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386440 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:20 tscc-2-50 kernel: Lustre: 2956:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386432/real 1408386432]  req@ffff881f71f4dc00 x1476518982737712/t0(0) o38->dolphin-MDT0000-mdc-ffff881065a37400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386440 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:20 tscc-1-51 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386432/real 1408386432]  req@ffff882025a86800 x1476542205929748/t0(0) o38->dolphin-MDT0000-mdc-ffff882025d49c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386440 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:21 tscc-1-9 kernel: Lustre: 2739:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386382/real 1408386382]  req@ffff8809d66c7000 x1476519677615820/t0(0) o400->dolphin-MDT0000-mdc-ffff8840101b8c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386441 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:21 tscc-1-9 kernel: Lustre: dolphin-MDT0000-mdc-ffff8840101b8c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:21 tscc-2-11 kernel: Lustre: 2967:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386379/real 1408386379]  req@ffff88105b8bdc00 x1476519079108188/t0(0) o400->dolphin-MDT0000-mdc-ffff8820686ba000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386441 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:21 tscc-2-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff8820686ba000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:21 tscc-gpu-3-4 kernel: Lustre: 18128:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386383/real 1408386383]  req@ffff88083ca2fc00 x1476571902028796/t0(0) o101->dolphin-MDT0000-mdc-ffff88087229d800@172.25.32.25@tcp:12/10 lens 376/1928 e 0 to 1 dl 1408386441 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 11:27:24 tscc-0-63 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386386/real 1408386386]  req@ffff8810323ad000 x1476798012478204/t0(0) o400->dolphin-MDT0000-mdc-ffff881031346c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386444 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:24 tscc-0-63 kernel: Lustre: dolphin-MDT0000-mdc-ffff881031346c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:26 tscc-1-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386437/real 1408386437]  req@ffff880d8a1e9800 x1476536988250964/t0(0) o38->dolphin-MDT0000-mdc-ffff881025a03800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386446 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:27 tscc-3-54 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386389/real 1408386389]  req@ffff880ea1e65400 x1476290785566564/t0(0) o400->dolphin-MDT0000-mdc-ffff881026a4a400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386447 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:27 tscc-3-54 kernel: Lustre: dolphin-MDT0000-mdc-ffff881026a4a400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:28 tscc-2-55 kernel: Lustre: 2969:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386390/real 1408386390]  req@ffff881fa46a6800 x1476519002406436/t0(0) o400->dolphin-MDT0000-mdc-ffff882068dc4c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386448 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:28 tscc-2-55 kernel: Lustre: dolphin-MDT0000-mdc-ffff882068dc4c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:27:28 tscc-1-9 kernel: Lustre: 2738:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386441/real 1408386441]  req@ffff8815ae02c800 x1476519677616348/t0(0) o38->dolphin-MDT0000-mdc-ffff8840101b8c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386448 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:28 tscc-0-0 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386448/real 1408386448]  req@ffff8810310a0000 x1476519380600616/t0(0) o38->dolphin-MDT0000-mdc-ffff8810326bb000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386454 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:28 tscc-0-39 kernel: Lustre: 2216:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386448/real 1408386448]  req@ffff880831f9d000 x1475608471412264/t0(0) o38->dolphin-MDT0000-mdc-ffff88081c7a2000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386454 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:29 tscc-gpu-5-7 kernel: Lustre: 30854:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386403/real 1408386403]  req@ffff8808677a2c00 x1476612406249480/t0(0) o35->dolphin-MDT0000-mdc-ffff880876ea6c00@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408386449 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 11:27:29 tscc-pdafm-7-0 kernel: Lustre: 3148:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386449/real 1408386449]  req@ffff88800056c000 x1476534244792528/t0(0) o38->dolphin-MDT0000-mdc-ffff885010079800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386455 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:29 tscc-0-18 kernel: Lustre: 2753:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386449/real 1408386449]  req@ffff88081f1d5400 x1476547249475204/t0(0) o38->dolphin-MDT0000-mdc-ffff8810343ce400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386455 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:29 tscc-0-13 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386449/real 1408386449]  req@ffff8809e5cb9800 x1476634825390888/t0(0) o38->dolphin-MDT0000-mdc-ffff880830ad3800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386455 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:29 tscc-3-49 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386449/real 1408386449]  req@ffff880d8451f400 x1476519923736440/t0(0) o38->dolphin-MDT0000-mdc-ffff882026177c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386455 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:29 tscc-2-37 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386449/real 1408386449]  req@ffff88081d770400 x1476570224060456/t0(0) o38->dolphin-MDT0000-mdc-ffff881033c5a800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386455 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:30 tscc-0-63 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386444/real 1408386444]  req@ffff881032cb2400 x1476798012478732/t0(0) o38->dolphin-MDT0000-mdc-ffff881031346c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386450 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:31 tscc-1-12 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386451/real 1408386451]  req@ffff88081b0b2800 x1476547036056412/t0(0) o38->dolphin-MDT0000-mdc-ffff882011047000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386457 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:31 tscc-2-11 kernel: Lustre: 2959:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386441/real 1408386441]  req@ffff88105b8bd400 x1476519079108716/t0(0) o38->dolphin-MDT0000-mdc-ffff8820686ba000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386451 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:32 tscc-0-60 kernel: Lustre: 2813:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386452/real 1408386452]  req@ffff88103301ec00 x1476700762674984/t0(0) o38->dolphin-MDT0000-mdc-ffff881030ca2400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386459 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:32 tscc-1-33 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386452/real 1408386452]  req@ffff880dace59800 x1476536972524316/t0(0) o38->dolphin-MDT0000-mdc-ffff8810264d1400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386459 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:33 tscc-gpu-5-2 kernel: Lustre: 2625:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386453/real 1408386453]  req@ffff880828acd400 x1476558499481596/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386464 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:33 tscc-gpu-5-2 kernel: Lustre: 2625:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Aug 18 11:27:33 tscc-3-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386447/real 1408386447]  req@ffff8803f0dc6400 x1476290785567092/t0(0) o38->dolphin-MDT0000-mdc-ffff881026a4a400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386453 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:33 tscc-1-46 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386453/real 1408386453]  req@ffff880f73305c00 x1476536973595276/t0(0) o38->dolphin-MDT0000-mdc-ffff88101d826800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386460 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:33 tscc-gpu-6-1 kernel: Lustre: 2523:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386435/real 1408386435]  req@ffff88080b961000 x1476558706387012/t0(0) o38->dolphin-MDT0000-mdc-ffff880476c19c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386453 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:33 tscc-gpu-6-1 kernel: Lustre: 2523:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:27:34 tscc-2-55 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386448/real 1408386448]  req@ffff881f0ec94000 x1476519002406964/t0(0) o38->dolphin-MDT0000-mdc-ffff882068dc4c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386454 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:34 tscc-1-4 kernel: Lustre: 2732:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386454/real 1408386454]  req@ffff880c419b7000 x1475918802709696/t0(0) o38->dolphin-MDT0000-mdc-ffff88101ea3f000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386460 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:34 tscc-0-21 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386448/real 1408386448]  req@ffff881033ef7c00 x1476699991532140/t0(0) o38->dolphin-MDT0000-mdc-ffff881032040400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386454 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:35 tscc-3-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386419/real 1408386419]  req@ffff880e5cc37c00 x1476519856702100/t0(0) o38->dolphin-MDT0000-mdc-ffff882026020800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386455 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:36 tscc-0-5 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386450/real 1408386450]  req@ffff880814c4f800 x1476638712319228/t0(0) o38->dolphin-MDT0000-mdc-ffff88103170a000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386456 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:37 tscc-2-50 kernel: Lustre: 2956:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386457/real 1408386457]  req@ffff881f71f4dc00 x1476518982737720/t0(0) o38->dolphin-MDT0000-mdc-ffff881065a37400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386465 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:37 tscc-0-26 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386457/real 1408386457]  req@ffff880fc0989800 x1476612864112196/t0(0) o38->dolphin-MDT0000-mdc-ffff8810314d8800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386464 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:37 tscc-gpu-6-6 kernel: Lustre: 2524:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386457/real 1408386457]  req@ffff88086a65f800 x1476558757918916/t0(0) o38->dolphin-MDT0000-mdc-ffff880878d95000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386464 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:37 tscc-0-8 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386457/real 1408386457]  req@ffff880814607c00 x1476547167666228/t0(0) o38->dolphin-MDT0000-mdc-ffff88082f334800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386464 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:38 tscc-1-43 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386458/real 1408386458]  req@ffff880d79283000 x1476570186887128/t0(0) o38->dolphin-MDT0000-mdc-ffff8810259ff400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386464 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:39 tscc-0-39 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386401/real 1408386401]  req@ffff8801569d0400 x1475608471411996/t0(0) o400->dolphin-MDT0000-mdc-ffff88081c7a2000@172.25.32.53@tcp:12/10 lens 224/224 e 0 to 1 dl 1408386459 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:39 tscc-1-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386453/real 1408386453]  req@ffff880edfae9400 x1476537045916924/t0(0) o38->dolphin-MDT0000-mdc-ffff882026991400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386459 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:39 tscc-gpu-3-2 kernel: Lustre: 2905:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386452/real 1408386452]  req@ffff88087a644000 x1476798148792036/t0(0) o38->dolphin-MDT0000-mdc-ffff8808767dd000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386459 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:40 tscc-gpu-3-4 kernel: Lustre: 2555:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386449/real 1408386449]  req@ffff88105784cc00 x1476571902029064/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386460 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:40 tscc-gpu-3-4 kernel: Lustre: 2555:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:27:40 tscc-1-59 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386453/real 1408386453]  req@ffff880522c52c00 x1476536863839976/t0(0) o38->dolphin-MDT0000-mdc-ffff880831706400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386460 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:44 tscc-0-29 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386457/real 1408386457]  req@ffff880831ff6c00 x1476612872576764/t0(0) o38->dolphin-MDT0000-mdc-ffff8808308f6c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386464 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:45 tscc-1-38 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386457/real 1408386457]  req@ffff880fe23a7c00 x1476575145559952/t0(0) o38->dolphin-MDT0000-mdc-ffff882026301000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386465 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:45 tscc-0-34 kernel: Lustre: 2756:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386457/real 1408386457]  req@ffff880796f93000 x1476614108115096/t0(0) o38->dolphin-MDT0000-mdc-ffff881030f1ac00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386465 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:45 tscc-1-51 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386457/real 1408386457]  req@ffff881d4d529800 x1476542205929756/t0(0) o38->dolphin-MDT0000-mdc-ffff882025d49c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386465 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:45 tscc-1-4 kernel: Lustre: 2732:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386454/real 1408386454]  req@ffff880bd72ea400 x1475918802709692/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386465 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:46 tscc-1-9 kernel: Lustre: 2738:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386466/real 1408386466]  req@ffff8815ae02c800 x1476519677616356/t0(0) o38->dolphin-MDT0000-mdc-ffff8840101b8c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386473 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:46 tscc-2-11 kernel: Lustre: 2959:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386466/real 1408386466]  req@ffff88105b8bd400 x1476519079108724/t0(0) o38->dolphin-MDT0000-mdc-ffff8820686ba000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386476 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:46 tscc-gpu-5-7 kernel: Lustre: 2551:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386423/real 1408386423]  req@ffff8804525c6c00 x1476612406249728/t0(0) o38->dolphin-MDT0000-mdc-ffff880876ea6c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386466 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:46 tscc-gpu-5-7 kernel: Lustre: 2551:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:27:51 tscc-1-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386462/real 1408386462]  req@ffff880f163dac00 x1476536988250972/t0(0) o38->dolphin-MDT0000-mdc-ffff881025a03800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386471 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:52 tscc-3-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386472/real 1408386472]  req@ffff8803f0dc6000 x1476290785567100/t0(0) o38->dolphin-MDT0000-mdc-ffff881026a4a400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386478 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:53 tscc-2-55 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386473/real 1408386473]  req@ffff881fa46a6800 x1476519002406972/t0(0) o38->dolphin-MDT0000-mdc-ffff882068dc4c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386479 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:53 tscc-0-21 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386473/real 1408386473]  req@ffff881033ef7400 x1476699991532400/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386484 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:53 tscc-0-21 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:27:55 tscc-0-63 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386469/real 1408386469]  req@ffff8810323adc00 x1476798012478740/t0(0) o38->dolphin-MDT0000-mdc-ffff881031346c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386475 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:55 tscc-0-5 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386475/real 1408386475]  req@ffff8808136efc00 x1476638712319488/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386486 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:55 tscc-0-5 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:27:58 tscc-1-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386478/real 1408386478]  req@ffff880edfae9800 x1476537045917184/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386489 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:58 tscc-1-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:27:58 tscc-1-59 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386478/real 1408386478]  req@ffff88081d8ebc00 x1476536863840236/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386490 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:27:58 tscc-1-59 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:02 tscc-1-38 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386482/real 1408386482]  req@ffff880c172ad800 x1476575145560212/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386495 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:02 tscc-1-38 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:02 tscc-0-34 kernel: Lustre: 2756:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386482/real 1408386482]  req@ffff880796f93800 x1476614108115356/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386496 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:02 tscc-0-34 kernel: Lustre: 2756:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:02 tscc-1-51 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386482/real 1408386482]  req@ffff881e372bd800 x1476542205930016/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386495 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:02 tscc-1-51 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:02 tscc-0-29 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386482/real 1408386482]  req@ffff88081132fc00 x1476612872577024/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386494 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:02 tscc-0-29 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:04 tscc-0-0 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386473/real 1408386473]  req@ffff8810310a0400 x1476519380600876/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386484 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:04 tscc-0-0 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:04 tscc-0-39 kernel: Lustre: 2216:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386473/real 1408386473]  req@ffff88071c966400 x1475608471412528/t0(0) o38->dolphin-MDT0000-mdc-ffff88081c7a2000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386484 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:04 tscc-0-39 kernel: Lustre: 2216:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:28:05 tscc-pdafm-7-0 kernel: Lustre: 3148:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386474/real 1408386474]  req@ffff888010c73800 x1476534244792792/t0(0) o38->dolphin-MDT0000-mdc-ffff885010079800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386485 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:05 tscc-pdafm-7-0 kernel: Lustre: 3148:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:05 tscc-0-18 kernel: Lustre: 2753:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386474/real 1408386474]  req@ffff880820b02000 x1476547249475468/t0(0) o38->dolphin-MDT0000-mdc-ffff8810343ce400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386485 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:05 tscc-0-18 kernel: Lustre: 2753:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:05 tscc-0-13 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386474/real 1408386474]  req@ffff880ac8359000 x1476634825391148/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386485 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:05 tscc-0-13 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:05 tscc-3-49 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386474/real 1408386474]  req@ffff880d8451f000 x1476519923736704/t0(0) o38->dolphin-MDT0000-mdc-ffff882026177c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386485 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:05 tscc-3-49 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:05 tscc-2-37 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386474/real 1408386474]  req@ffff880188369000 x1476570224060720/t0(0) o38->dolphin-MDT0000-mdc-ffff881033c5a800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386485 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:05 tscc-2-37 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:07 tscc-1-12 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386476/real 1408386476]  req@ffff8819db253c00 x1476547036056676/t0(0) o38->dolphin-MDT0000-mdc-ffff882011047000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386487 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:07 tscc-1-12 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:09 tscc-0-60 kernel: Lustre: 2813:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386477/real 1408386477]  req@ffff8810331f8800 x1476700762675244/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386489 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:09 tscc-0-60 kernel: Lustre: 2813:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:09 tscc-1-33 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386477/real 1408386477]  req@ffff88102582a000 x1476536972524580/t0(0) o38->dolphin-MDT0000-mdc-ffff8810264d1400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386489 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:09 tscc-1-33 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:09 tscc-gpu-3-2 kernel: Lustre: 2905:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386477/real 1408386477]  req@ffff880868f33400 x1476798148792300/t0(0) o38->dolphin-MDT0000-mdc-ffff8808767dd000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386489 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:09 tscc-gpu-3-2 kernel: Lustre: 2905:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:10 tscc-1-46 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386478/real 1408386478]  req@ffff880f73305c00 x1476536973595540/t0(0) o38->dolphin-MDT0000-mdc-ffff88101d826800@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386490 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:10 tscc-1-46 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:10 tscc-1-4 kernel: Lustre: 2732:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386479/real 1408386479]  req@ffff880ce9582400 x1475918802709960/t0(0) o38->dolphin-MDT0000-mdc-ffff88101ea3f000@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386490 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:10 tscc-1-4 kernel: Lustre: 2732:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:13 tscc-gpu-5-2 kernel: Lustre: 2625:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386478/real 1408386478]  req@ffff8807d7646400 x1476558499481864/t0(0) o38->dolphin-MDT0000-mdc-ffff88086f941c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386493 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:13 tscc-gpu-5-2 kernel: Lustre: 2625:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 11:28:13 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386477/real 1408386477]  req@ffff880f2784b400 x1476519914724148/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386493 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:13 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 11:28:14 tscc-1-43 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386483/real 1408386483]  req@ffff88100d2e1c00 x1476570186887392/t0(0) o38->dolphin-MDT0000-mdc-ffff8810259ff400@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386494 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:14 tscc-1-43 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:14 tscc-0-26 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386482/real 1408386482]  req@ffff880fc0989c00 x1476612864112456/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386494 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:14 tscc-0-26 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:14 tscc-gpu-6-6 kernel: Lustre: 2524:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386482/real 1408386482]  req@ffff88086bf30000 x1476558757919176/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386494 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:14 tscc-gpu-6-6 kernel: Lustre: 2524:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:14 tscc-0-8 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386482/real 1408386482]  req@ffff8808317e9400 x1476547167666488/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386494 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:14 tscc-0-8 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:15 tscc-2-50 kernel: Lustre: 2956:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386482/real 1408386482]  req@ffff881f71f4d400 x1476518982737980/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386495 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:15 tscc-2-50 kernel: Lustre: 2956:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:19 tscc-gpu-3-4 kernel: Lustre: 2555:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386499/real 1408386499]  req@ffff88105e6d3400 x1476571902029596/t0(0) o38->dolphin-MDT0000-mdc-ffff88087229d800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386510 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:19 tscc-gpu-3-4 kernel: Lustre: 2555:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:23 tscc-1-9 kernel: Lustre: 2738:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386491/real 1408386491]  req@ffff8815ae02c400 x1476519677616616/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386503 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:23 tscc-1-9 kernel: Lustre: 2738:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:23 tscc-gpu-6-1 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-gpu-6-1 kernel: Lustre: Evicted from dolphin-MDT0000_UUID (at 172.25.32.25@tcp) after server handle changed from 0x30845f552b695421 to 0x30845f5538be2f28
Aug 18 11:28:23 tscc-gpu-6-1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880476c19c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:28:23 tscc-3-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026020800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-gpu-6-1 kernel: LustreError: 2591:0:(file.c:171:ll_close_inode_openhandle()) inode 144115537243275894 mdc close failed: rc = -5
Aug 18 11:28:23 tscc-gpu-6-1 kernel: LustreError: 2587:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:28:23 tscc-gpu-6-1 kernel: LustreError: 2587:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 196 previous similar messages
Aug 18 11:28:23 tscc-gpu-6-1 kernel: LustreError: 31619:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff880476c19c00: Can't allocate new meta-sequence,rc -108
Aug 18 11:28:23 tscc-gpu-6-1 kernel: LustreError: 31619:0:(fid_request.c:220:seq_client_alloc_seq()) Skipped 1 previous similar message
Aug 18 11:28:23 tscc-gpu-6-1 kernel: LustreError: 31619:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff880476c19c00: Can't allocate new sequence, rc -108
Aug 18 11:28:23 tscc-gpu-6-1 kernel: LustreError: 31619:0:(fid_request.c:354:seq_client_alloc_fid()) Skipped 1 previous similar message
Aug 18 11:28:23 tscc-gpu-6-1 kernel: LustreError: 31619:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 11:28:23 tscc-gpu-6-1 kernel: LustreError: 31619:0:(mdc_locks.c:1209:mdc_intent_lock()) Skipped 1 previous similar message
Aug 18 11:28:23 tscc-gpu-6-1 kernel: LustreError: 2591:0:(file.c:171:ll_close_inode_openhandle()) Skipped 1 previous similar message
Aug 18 11:28:23 tscc-2-11 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8820686ba000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 11:28:23 tscc-gpu-3-4 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-3-54 kernel: Lustre: dolphin-MDT0000-mdc-ffff881026a4a400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-0-21 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-2-37 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-0-63 kernel: Lustre: dolphin-MDT0000-mdc-ffff881031346c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-gpu-3-2 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-0-13 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-0-60 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-gpu-6-1 kernel: LustreError: 2539:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x20000514c:0x739:0x0] error -108.
Aug 18 11:28:23 tscc-gpu-6-1 kernel: LustreError: 2539:0:(vvp_io.c:1229:vvp_io_init()) Skipped 251 previous similar messages
Aug 18 11:28:23 tscc-0-5 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-2-55 kernel: Lustre: dolphin-MDT0000-mdc-ffff882068dc4c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-pdafm-7-0 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-0-0 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-1-46 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-0-18 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-3-46 kernel: Lustre: dolphin-MDT0000-mdc-ffff882028f52400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-1-33 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-gpu-5-7 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-0-39 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-1-12 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-3-49 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-1-41 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:23 tscc-1-59 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:24 tscc-1-4 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:25 tscc-0-63 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386494/real 1408386494]  req@ffff8810323ad800 x1476798012479000/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386505 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:25 tscc-0-63 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:26 tscc-2-11 kernel: Lustre: 2959:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386491/real 1408386491]  req@ffff88105b8bd800 x1476519079108984/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386506 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:26 tscc-2-11 kernel: Lustre: 2959:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:27 tscc-1-38 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:27 tscc-0-34 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:27 tscc-2-50 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:27 tscc-1-51 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:27 tscc-0-26 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:27 tscc-gpu-6-6 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:27 tscc-0-29 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:27 tscc-0-8 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:28 tscc-3-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386497/real 1408386497]  req@ffff8803f0dc6400 x1476290785567360/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386508 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:28 tscc-3-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:28 tscc-1-43 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:29 tscc-2-55 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386498/real 1408386498]  req@ffff881d7cbb9000 x1476519002407232/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386509 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:29 tscc-2-55 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:32 tscc-1-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386512/real 1408386512]  req@ffff880df4832800 x1476536988251500/t0(0) o38->dolphin-MDT0000-mdc-ffff881025a03800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386526 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:32 tscc-1-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Aug 18 11:28:32 tscc-1-54 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:35 tscc-gpu-5-3 kernel: Lustre: 29761:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386424/real 1408386424]  req@ffff88086ee4fc00 x1476570129714104/t0(0) o101->dolphin-MDT0000-mdc-ffff880876924800@172.25.32.25@tcp:12/10 lens 616/1136 e 1 to 1 dl 1408386515 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1
Aug 18 11:28:35 tscc-gpu-5-3 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876924800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:28:35 tscc-gpu-5-3 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876924800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 11:28:35 tscc-gpu-5-3 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 11:28:36 tscc-1-9 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:36 tscc-2-11 kernel: Lustre: dolphin-MDT0000-mdc-ffff8820686ba000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:36 tscc-gpu-5-4 kernel: Lustre: 30063:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386425/real 1408386425]  req@ffff880455af9000 x1476612337940476/t0(0) o101->dolphin-MDT0000-mdc-ffff880877501800@172.25.32.25@tcp:12/10 lens 616/1136 e 1 to 1 dl 1408386516 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1
Aug 18 11:28:36 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:28:36 tscc-gpu-5-4 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880877501800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 11:28:36 tscc-gpu-5-7 kernel: Lustre: 2551:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386473/real 1408386473]  req@ffff880433454000 x1476612406250012/t0(0) o38->dolphin-MDT0000-mdc-ffff880876ea6c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386516 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:36 tscc-gpu-5-7 kernel: Lustre: 2551:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:28:36 tscc-login1 kernel: Lustre: 56313:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386425/real 1408386425]  req@ffff88094f1e2c00 x1476710235088136/t0(0) o101->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.25@tcp:12/10 lens 616/1136 e 1 to 1 dl 1408386516 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1
Aug 18 11:28:36 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:28:36 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 11:28:36 tscc-login1 kernel: LustreError: Skipped 1 previous similar message
Aug 18 11:28:38 tscc-gpu-6-4 kernel: Lustre: 16082:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386424/real 1408386424]  req@ffff88072f735000 x1476570629144224/t0(0) o101->dolphin-MDT0000-mdc-ffff880879578000@172.25.32.25@tcp:12/10 lens 640/1136 e 1 to 1 dl 1408386518 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1
Aug 18 11:28:38 tscc-gpu-6-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880879578000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:28:38 tscc-gpu-6-4 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880879578000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 11:28:38 tscc-gpu-6-4 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 11:28:39 tscc-0-63 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:40 tscc-1-38 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386507/real 1408386507]  req@ffff880cb53cbc00 x1476575145560480/t0(0) o38->dolphin-MDT0000-mdc-ffff882026301000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386520 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:40 tscc-1-38 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:28:40 tscc-1-51 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386507/real 1408386507]  req@ffff881c0e525000 x1476542205930284/t0(0) o38->dolphin-MDT0000-mdc-ffff882025d49c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386520 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:28:40 tscc-1-51 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:28:42 tscc-3-54 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:43 tscc-2-55 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:43 tscc-0-0 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810326bb000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:43 tscc-0-39 kernel: Lustre: dolphin-MDT0000-mdc-ffff88081c7a2000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:43 tscc-gpu-5-7 kernel: Lustre: Evicted from dolphin-MDT0000_UUID (at 172.25.32.25@tcp) after server handle changed from 0x30845f552b0c58a8 to 0x30845f5538be3349
Aug 18 11:28:43 tscc-gpu-5-7 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880876ea6c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:28:43 tscc-gpu-5-7 kernel: LustreError: 30854:0:(file.c:171:ll_close_inode_openhandle()) inode 144115536974841501 mdc close failed: rc = -5
Aug 18 11:28:43 tscc-gpu-5-7 kernel: LustreError: 30623:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:28:43 tscc-gpu-5-7 kernel: LustreError: 30623:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 44 previous similar messages
Aug 18 11:28:43 tscc-gpu-5-7 kernel: LustreError: 30623:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x20000513c:0x7f7:0x0] error -5.
Aug 18 11:28:43 tscc-gpu-5-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876ea6c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:43 tscc-0-21 kernel: Lustre: dolphin-MDT0000-mdc-ffff881032040400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:44 tscc-gpu-3-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff88087229d800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:44 tscc-pdafm-7-0 kernel: Lustre: dolphin-MDT0000-mdc-ffff885010079800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:44 tscc-0-18 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810343ce400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:44 tscc-0-13 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830ad3800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:44 tscc-3-49 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026177c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:44 tscc-2-37 kernel: Lustre: dolphin-MDT0000-mdc-ffff881033c5a800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:45 tscc-0-5 kernel: Lustre: dolphin-MDT0000-mdc-ffff88103170a000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:46 tscc-1-12 kernel: Lustre: dolphin-MDT0000-mdc-ffff882011047000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:47 tscc-0-60 kernel: Lustre: dolphin-MDT0000-mdc-ffff881030ca2400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:47 tscc-3-46 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:47 tscc-1-33 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810264d1400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:47 tscc-gpu-3-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808767dd000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:48 tscc-gpu-5-2 kernel: Lustre: Evicted from dolphin-MDT0000_UUID (at 172.25.32.25@tcp) after server handle changed from 0x30845f552b0ca649 to 0x30845f5538be2f8a
Aug 18 11:28:48 tscc-gpu-5-2 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff88086f941c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:28:48 tscc-gpu-5-2 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:28:48 tscc-gpu-5-2 kernel: LustreError: 28087:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:28:48 tscc-gpu-5-2 kernel: LustreError: 28087:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 28 previous similar messages
Aug 18 11:28:48 tscc-gpu-5-2 kernel: LustreError: 27869:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x20000513c:0x81d:0x0] error -5.
Aug 18 11:28:48 tscc-gpu-5-2 kernel: LustreError: 2083:0:(file.c:171:ll_close_inode_openhandle()) inode 144115536991617189 mdc close failed: rc = -108
Aug 18 11:28:48 tscc-gpu-5-2 kernel: LustreError: 28127:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff88086f941c00: Can't allocate new meta-sequence,rc -108
Aug 18 11:28:48 tscc-gpu-5-2 kernel: LustreError: 28127:0:(fid_request.c:220:seq_client_alloc_seq()) Skipped 1 previous similar message
Aug 18 11:28:48 tscc-gpu-5-2 kernel: LustreError: 28127:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff88086f941c00: Can't allocate new sequence, rc -108
Aug 18 11:28:48 tscc-gpu-5-2 kernel: LustreError: 28127:0:(fid_request.c:354:seq_client_alloc_fid()) Skipped 1 previous similar message
Aug 18 11:28:48 tscc-gpu-5-2 kernel: LustreError: 28127:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 11:28:48 tscc-gpu-5-2 kernel: LustreError: 28127:0:(mdc_locks.c:1209:mdc_intent_lock()) Skipped 1 previous similar message
Aug 18 11:28:48 tscc-gpu-5-2 kernel: LustreError: 2083:0:(ldlm_resource.c:809:ldlm_resource_complain()) dolphin-MDT0000-mdc-ffff88086f941c00: namespace resource [0x20000513d:0x9e5:0x0].0 (ffff88086c51c800) refcount nonzero (1) after lock cleanup; forcing cleanup.
Aug 18 11:28:48 tscc-gpu-5-2 kernel: LustreError: 2083:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x20000513d:0x9e5:0x0].0 (ffff88086c51c800) refcount = 2
Aug 18 11:28:48 tscc-1-46 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101d826800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:48 tscc-1-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026991400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:48 tscc-1-59 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831706400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:49 tscc-1-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101ea3f000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:52 tscc-1-38 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026301000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:52 tscc-0-34 kernel: Lustre: dolphin-MDT0000-mdc-ffff881030f1ac00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:52 tscc-2-50 kernel: Lustre: dolphin-MDT0000-mdc-ffff881065a37400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:52 tscc-1-51 kernel: Lustre: dolphin-MDT0000-mdc-ffff882025d49c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:52 tscc-0-26 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810314d8800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:52 tscc-gpu-6-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff880878d95000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:52 tscc-0-29 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808308f6c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:52 tscc-0-8 kernel: Lustre: dolphin-MDT0000-mdc-ffff88082f334800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:53 tscc-1-43 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810259ff400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:28:57 tscc-1-54 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025a03800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:29:00 tscc-gpu-5-3 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876924800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:29:01 tscc-1-9 kernel: Lustre: dolphin-MDT0000-mdc-ffff8840101b8c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:29:01 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:29:01 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:29:03 tscc-gpu-6-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880879578000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:29:44 tscc-1-40 kernel: Lustre: 2736:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386577/real 1408386577]  req@ffff88102820d000 x1476537020780088/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386584 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:29:44 tscc-1-40 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:29:45 tscc-gpu-5-4 kernel: Lustre: 30722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386574/real 1408386574]  req@ffff88080b8c3400 x1476612337949500/t0(0) o35->dolphin-MDT0000-mdc-ffff880877501800@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408386585 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 11:29:45 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:29:46 tscc-gpu-3-1 kernel: Lustre: 2918:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386579/real 1408386579]  req@ffff88084458cc00 x1476798077491236/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386586 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:29:46 tscc-gpu-3-1 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:29:47 tscc-3-56 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386580/real 1408386580]  req@ffff881033d5ec00 x1476519829364284/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386587 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:29:47 tscc-3-56 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:29:47 tscc-1-32 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386580/real 1408386580]  req@ffff8820272de800 x1476439898265980/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386587 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:29:47 tscc-1-32 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:29:47 tscc-login1 kernel: Lustre: 2822:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386577/real 1408386577]  req@ffff880719a5cc00 x1476710235091432/t0(0) o103->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.25@tcp:17/18 lens 344/224 e 0 to 1 dl 1408386587 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 11:29:47 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:29:48 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 11:29:48 tscc-gpu-5-4 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880877501800: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 11:29:48 tscc-1-40 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:29:48 tscc-gpu-3-1 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:29:48 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:29:48 tscc-login1 kernel: LustreError: 56313:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:29:48 tscc-login1 kernel: LustreError: 55183:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x20000518a:0x16c:0x0] error -5.
Aug 18 11:29:48 tscc-login1 kernel: LustreError: 55183:0:(vvp_io.c:1229:vvp_io_init()) Skipped 2 previous similar messages
Aug 18 11:29:48 tscc-login1 kernel: LustreError: 56313:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 3 previous similar messages
Aug 18 11:29:48 tscc-gpu-5-4 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880877501800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:29:48 tscc-3-56 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:29:48 tscc-1-32 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:29:48 tscc-gpu-5-4 kernel: LustreError: 30722:0:(file.c:171:ll_close_inode_openhandle()) inode 144115537612374858 mdc close failed: rc = -5
Aug 18 11:29:48 tscc-gpu-5-4 kernel: LustreError: 27205:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:29:48 tscc-gpu-5-4 kernel: LustreError: 30788:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:29:48 tscc-gpu-5-4 kernel: LustreError: 30788:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 1 previous similar message
Aug 18 11:29:48 tscc-gpu-5-4 kernel: LustreError: 27205:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005162:0x353:0x0] error -5.
Aug 18 11:29:48 tscc-gpu-5-4 kernel: LustreError: 30672:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID [0x200005162:0x367:0x0] error: rc = -108
Aug 18 11:29:48 tscc-gpu-5-4 kernel: LustreError: 30672:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 1 previous similar message
Aug 18 11:29:48 tscc-gpu-5-4 kernel: LustreError: 30786:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff880877501800: Can't allocate new meta-sequence,rc -108
Aug 18 11:29:48 tscc-gpu-5-4 kernel: LustreError: 30786:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff880877501800: Can't allocate new sequence, rc -108
Aug 18 11:29:48 tscc-gpu-5-4 kernel: LustreError: 30786:0:(mdc_reint.c:235:mdc_create()) Can't alloc new fid, rc -108
Aug 18 11:29:48 tscc-gpu-5-4 kernel: LustreError: 30722:0:(file.c:171:ll_close_inode_openhandle()) Skipped 5 previous similar messages
Aug 18 11:29:48 tscc-login1 kernel: LustreError: 56313:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID [0x250029c7:0x87004be5:0x0] error: rc = -108
Aug 18 11:29:48 tscc-login1 kernel: LustreError: 62836:0:(file.c:171:ll_close_inode_openhandle()) inode 144115538249908236 mdc close failed: rc = -108
Aug 18 11:29:48 tscc-login1 kernel: LustreError: 62836:0:(file.c:171:ll_close_inode_openhandle()) Skipped 25 previous similar messages
Aug 18 11:29:48 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:29:48 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:29:56 tscc-gpu-6-3 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 11:30:01 tscc-gpu-6-3 kernel: Lustre: 2936:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386596/real 1408386596]  req@ffff88047084ec00 x1476800383287300/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386601 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:30:02 tscc-gpu-6-3 kernel: LustreError: 2872:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88047084e800 x1476800383287304/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 11:30:09 tscc-gpu-6-3 kernel: LustreError: 2966:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88047084e000 x1476800383287312/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 11:30:26 tscc-gpu-6-3 kernel: Lustre: 2936:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386621/real 1408386621]  req@ffff880478dcec00 x1476800383287396/t0(0) o38->dolphin-MDT0000-mdc-ffff88047735d800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386626 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:30:46 tscc-gpu-6-3 kernel: Lustre: Layout lock feature supported.
Aug 18 11:30:47 tscc-gpu-6-3 kernel: Lustre: Mounted dolphin-client
Aug 18 11:31:59 tscc-3-56 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386712/real 1408386712]  req@ffff881032a90400 x1476519829365640/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386719 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:31:59 tscc-3-56 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:31:59 tscc-1-32 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386712/real 1408386712]  req@ffff882022821000 x1476439898267336/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386719 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:31:59 tscc-1-32 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:31:59 tscc-gpu-3-1 kernel: Lustre: 2916:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386711/real 1408386711]  req@ffff880476bc1000 x1476798077492592/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386719 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:31:59 tscc-gpu-3-1 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:00 tscc-0-7 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386713/real 1408386713]  req@ffff88081b834400 x1476547271417192/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386720 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:00 tscc-0-7 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:00 tscc-1-35 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386713/real 1408386713]  req@ffff880f9be7f000 x1476285713832012/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386720 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:00 tscc-1-35 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:00 tscc-1-53 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386713/real 1408386713]  req@ffff880f83acb800 x1476537036508332/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386720 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:00 tscc-1-53 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:00 tscc-1-1 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386711/real 1408386711]  req@ffff880c81930800 x1476519531683988/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386720 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:00 tscc-1-1 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:00 tscc-login1 kernel: Lustre: 2822:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386712/real 1408386712]  req@ffff880103f85000 x1476710235101756/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386720 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:00 tscc-login1 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:00 tscc-3-40 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386711/real 1408386711]  req@ffff880d25c0d800 x1476520324395524/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386720 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:00 tscc-3-40 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:00 tscc-1-56 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386711/real 1408386711]  req@ffff881017711400 x1476536956416184/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386720 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:00 tscc-1-56 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:01 tscc-1-45 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386714/real 1408386714]  req@ffff882029378000 x1476537015522100/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386721 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:01 tscc-1-45 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:02 tscc-1-14 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386715/real 1408386715]  req@ffff8840126f9800 x1476519630577776/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386722 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:02 tscc-1-14 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:03 tscc-2-39 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386716/real 1408386716]  req@ffff88080e1fcc00 x1476547206169108/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386723 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:03 tscc-2-39 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:03 tscc-0-62 kernel: Lustre: 2470:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386716/real 1408386716]  req@ffff88083358fc00 x1475809553573064/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386723 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:03 tscc-0-62 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:03 tscc-1-11 kernel: Lustre: 2766:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386716/real 1408386716]  req@ffff880d702ec400 x1476519742458396/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386723 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:03 tscc-1-11 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:03 tscc-0-20 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386716/real 1408386716]  req@ffff880816675c00 x1476638723855604/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386723 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:03 tscc-0-20 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:03 tscc-0-57 kernel: Lustre: 2234:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386716/real 1408386716]  req@ffff880dcb78c800 x1475562021687228/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386723 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:03 tscc-0-57 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:03 tscc-gpu-7-7 kernel: Lustre: 2547:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386716/real 1408386716]  req@ffff88045abddc00 x1476536280060628/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386723 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:03 tscc-gpu-7-7 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:03 tscc-1-61 kernel: Lustre: 2744:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386716/real 1408386716]  req@ffff880c535ac800 x1476536929575736/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386723 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:03 tscc-1-61 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:03 tscc-2-13 kernel: Lustre: 2979:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386716/real 1408386716]  req@ffff881065853c00 x1476519055845740/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386723 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:03 tscc-2-13 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:04 tscc-2-52 kernel: Lustre: 2959:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386717/real 1408386717]  req@ffff88202d2ca400 x1476290267160436/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386724 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:04 tscc-2-52 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:04 tscc-0-15 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386717/real 1408386717]  req@ffff880820ad0800 x1476612877340576/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386724 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:04 tscc-0-15 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:05 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386719/real 1408386719]  req@ffff8810318b6000 x1476519829365904/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386725 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:05 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386719/real 1408386719]  req@ffff881e0ea41000 x1476439898267600/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386725 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:06 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386720/real 1408386720]  req@ffff88081be19800 x1476547271417456/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386726 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:06 tscc-1-35 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386720/real 1408386720]  req@ffff8809f2e49800 x1476285713832276/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386726 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:06 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386720/real 1408386720]  req@ffff88102879f800 x1476537036508596/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386726 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:06 tscc-gpu-3-1 kernel: Lustre: 2906:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386719/real 1408386719]  req@ffff880466c7d400 x1476798077492856/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386726 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:07 tscc-1-45 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386721/real 1408386721]  req@ffff881fefb01000 x1476537015522364/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386727 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:07 tscc-2-10 kernel: Lustre: 2968:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386720/real 1408386720]  req@ffff880a35cd5000 x1476519079960952/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386727 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:07 tscc-2-10 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:08 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386720/real 1408386720]  req@ffff880ebfeda800 x1476519531684264/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386728 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:08 tscc-3-40 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386720/real 1408386720]  req@ffff881010b9a800 x1476520324395788/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386728 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:08 tscc-1-14 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386722/real 1408386722]  req@ffff8840106dfc00 x1476519630578040/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386728 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:08 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386720/real 1408386720]  req@ffff8810149acc00 x1476536956416448/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386728 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:09 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386723/real 1408386723]  req@ffff8804cdc0ac00 x1476547206169372/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386729 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:09 tscc-0-62 kernel: Lustre: 2458:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386723/real 1408386723]  req@ffff8804c2b90000 x1475809553573328/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386729 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:09 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386723/real 1408386723]  req@ffff8802b6e45800 x1476519742458420/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386729 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:09 tscc-0-20 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386723/real 1408386723]  req@ffff880831f5cc00 x1476638723855868/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386729 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:09 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386723/real 1408386723]  req@ffff881031858800 x1475562021687492/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386729 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:09 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386723/real 1408386723]  req@ffff88044b6be000 x1476536280060892/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386729 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:09 tscc-gpu-6-3 kernel: Lustre: 2937:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386721/real 1408386721]  req@ffff88046e04e000 x1476800383288208/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386729 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:09 tscc-gpu-6-3 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:09 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386723/real 1408386723]  req@ffff880ce329e000 x1476536929576000/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386729 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:09 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386723/real 1408386723]  req@ffff8801afacb000 x1476519055846004/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386729 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:10 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386724/real 1408386724]  req@ffff881ef34c2400 x1476290267160700/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386730 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:10 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386724/real 1408386724]  req@ffff8803cff0b800 x1476612877340840/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386730 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:12 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 11:32:12 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880877501800: Communicating with 172.25.32.25@tcp, operation mds_sync failed with -107.
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: Skipped 5 previous similar messages
Aug 18 11:32:12 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:32:12 tscc-2-10 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:32:12 tscc-gpu-6-3 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:32:12 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880877501800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 27205:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 29533:0:(file.c:171:ll_close_inode_openhandle()) inode 144115538333794306 mdc close failed: rc = -5
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 29533:0:(file.c:171:ll_close_inode_openhandle()) Skipped 9 previous similar messages
Aug 18 11:32:12 tscc-login1 kernel: LustreError: 55183:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:32:12 tscc-login1 kernel: LustreError: 55183:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 10 previous similar messages
Aug 18 11:32:12 tscc-login1 kernel: LustreError: 55183:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x20000518e:0x2f:0x0] error -5.
Aug 18 11:32:12 tscc-login1 kernel: LustreError: 56313:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff8810311a7800: Can't allocate new meta-sequence,rc -108
Aug 18 11:32:12 tscc-login1 kernel: LustreError: 56313:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff8810311a7800: Can't allocate new sequence, rc -108
Aug 18 11:32:12 tscc-login1 kernel: LustreError: 56313:0:(mdc_reint.c:235:mdc_create()) Can't alloc new fid, rc -108
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 30456:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff880877501800: Can't allocate new meta-sequence,rc -108
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 30456:0:(fid_request.c:220:seq_client_alloc_seq()) Skipped 1 previous similar message
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 30456:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff880877501800: Can't allocate new sequence, rc -108
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 30456:0:(fid_request.c:354:seq_client_alloc_fid()) Skipped 1 previous similar message
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 30456:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 27205:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 109 previous similar messages
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 27205:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005136:0x5a4:0x0] error -5.
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 27205:0:(vvp_io.c:1229:vvp_io_init()) Skipped 26 previous similar messages
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 29533:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID [0x14e00001:0xa0252bbc:0x0] error: rc = -108
Aug 18 11:32:12 tscc-gpu-5-4 kernel: LustreError: 29533:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 4 previous similar messages
Aug 18 11:32:12 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:32:12 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:32:30 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386744/real 1408386744]  req@ffff8810341c1800 x1476519829365908/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386750 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:30 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386744/real 1408386744]  req@ffff881e0ea41000 x1476439898267604/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386750 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:31 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386745/real 1408386745]  req@ffff88081b834400 x1476547271417460/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386751 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:31 tscc-1-35 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386745/real 1408386745]  req@ffff880f9be7f000 x1476285713832280/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386751 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:31 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386745/real 1408386745]  req@ffff880f83acb000 x1476537036508600/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386751 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:31 tscc-gpu-3-1 kernel: Lustre: 2906:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386744/real 1408386744]  req@ffff880878f1cc00 x1476798077492860/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386751 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:32 tscc-1-45 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386746/real 1408386746]  req@ffff881ca8183800 x1476537015522368/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386752 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:32 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386745/real 1408386745]  req@ffff8804417fc800 x1476710235102868/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386752 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:32 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:32:33 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386745/real 1408386745]  req@ffff880855ab5800 x1476519531684836/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386753 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:33 tscc-3-40 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386745/real 1408386745]  req@ffff880cac9ad000 x1476520324395792/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386753 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:33 tscc-1-14 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386747/real 1408386747]  req@ffff884000664000 x1476519630578044/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386753 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:33 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386745/real 1408386745]  req@ffff881017711000 x1476536956416452/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386753 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:34 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386748/real 1408386748]  req@ffff88080e1fc000 x1476547206169376/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386754 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:34 tscc-0-62 kernel: Lustre: 2458:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386748/real 1408386748]  req@ffff880478673400 x1475809553573332/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386754 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:34 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386748/real 1408386748]  req@ffff8806f6b29c00 x1476519742459000/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386754 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:34 tscc-0-20 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386748/real 1408386748]  req@ffff880816c96c00 x1476638723855872/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386754 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:34 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386748/real 1408386748]  req@ffff8810315a0800 x1475562021687496/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386754 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:34 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386748/real 1408386748]  req@ffff88045abdd800 x1476536280060896/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386754 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:34 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386748/real 1408386748]  req@ffff880da2392400 x1476536929576004/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386754 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:34 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386748/real 1408386748]  req@ffff881ccbb15800 x1476519055846008/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386754 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:35 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386749/real 1408386749]  req@ffff88205272f000 x1476290267160704/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386755 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:35 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386749/real 1408386749]  req@ffff8806e7c8cc00 x1476612877340844/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386755 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:49 tscc-0-31 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386756/real 1408386756]  req@ffff880812e8b000 x1476612857990476/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386769 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:49 tscc-0-31 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:32:49 tscc-0-28 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386756/real 1408386756]  req@ffff88103203cc00 x1476547206508440/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386769 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:32:49 tscc-0-28 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:33:00 tscc-3-56 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386769/real 1408386769]  req@ffff881032e01400 x1476519829366172/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386780 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:00 tscc-1-32 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386769/real 1408386769]  req@ffff881e0ea41000 x1476439898267868/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386780 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:01 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386769/real 1408386769]  req@ffff88083089e400 x1476612857990740/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386781 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:01 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386770/real 1408386770]  req@ffff88081b834800 x1476547271417724/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386781 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:01 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386769/real 1408386769]  req@ffff880ff7fb3c00 x1476547206508704/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386781 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:01 tscc-1-35 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386770/real 1408386770]  req@ffff880e999e8000 x1476285713832544/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386781 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:01 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386770/real 1408386770]  req@ffff880f6147ec00 x1476537036508864/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386781 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:01 tscc-gpu-3-1 kernel: Lustre: 2906:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386769/real 1408386769]  req@ffff880872f19c00 x1476798077493124/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386781 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:02 tscc-1-45 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386771/real 1408386771]  req@ffff881ca8183800 x1476537015522632/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386782 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:03 tscc-1-1 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386770/real 1408386770]  req@ffff880f7f8b8800 x1476519531684868/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386783 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:03 tscc-3-40 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386770/real 1408386770]  req@ffff880cac9ad000 x1476520324396056/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386783 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:03 tscc-1-14 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386772/real 1408386772]  req@ffff884011466400 x1476519630578308/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386783 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:03 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386770/real 1408386770]  req@ffff881014a0b800 x1476536956416716/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386783 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:04 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386773/real 1408386773]  req@ffff880472ce5000 x1476547206169640/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386784 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:04 tscc-gpu-5-4 kernel: Lustre: 2623:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386757/real 1408386757]  req@ffff88045d830800 x1476612337978364/t0(0) o103->dolphin-MDT0000-mdc-ffff880877501800@172.25.32.25@tcp:17/18 lens 328/224 e 0 to 1 dl 1408386784 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 11:33:04 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:33:04 tscc-0-62 kernel: Lustre: 2458:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386773/real 1408386773]  req@ffff88049c677400 x1475809553573596/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386784 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:04 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386773/real 1408386773]  req@ffff881010a78400 x1476519742459004/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386784 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:04 tscc-0-20 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386773/real 1408386773]  req@ffff8808169a3c00 x1476638723856136/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386784 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:04 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386773/real 1408386773]  req@ffff8809bc1a8400 x1475562021687760/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386784 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:04 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386773/real 1408386773]  req@ffff88045abdd800 x1476536280061160/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386784 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:04 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386773/real 1408386773]  req@ffff880bbde78800 x1476536929576268/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386784 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:04 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386773/real 1408386773]  req@ffff881ccbb15800 x1476519055846272/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386784 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:05 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386774/real 1408386774]  req@ffff88205272f000 x1476290267160968/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386785 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:05 tscc-0-15 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386774/real 1408386774]  req@ffff880829af1800 x1476612877341108/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386785 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:15 tscc-0-7 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386795/real 1408386795]  req@ffff88081ca91c00 x1476547271417988/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:15 tscc-1-35 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386795/real 1408386795]  req@ffff8808e456b000 x1476285713832808/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:15 tscc-1-53 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386795/real 1408386795]  req@ffff880f6147ec00 x1476537036509128/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:15 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386795/real 1408386795]  req@ffff88053edcbc00 x1476710235103596/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386807 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:15 tscc-login1 kernel: Lustre: 2815:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:33:15 tscc-1-56 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386795/real 1408386795]  req@ffff881014a0b000 x1476536956416980/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386808 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:18 tscc-2-39 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386798/real 1408386798]  req@ffff8804de845c00 x1476547206169904/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:18 tscc-0-62 kernel: Lustre: 2458:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386798/real 1408386798]  req@ffff88083358fc00 x1475809553573860/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:18 tscc-gpu-5-4 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880877501800: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 11:33:18 tscc-gpu-5-4 kernel: LustreError: Skipped 10 previous similar messages
Aug 18 11:33:18 tscc-1-11 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386798/real 1408386798]  req@ffff88068d732c00 x1476519742459520/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:18 tscc-0-20 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386798/real 1408386798]  req@ffff8808169a3400 x1476638723856400/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:18 tscc-0-57 kernel: Lustre: 2231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386798/real 1408386798]  req@ffff8810314e1800 x1475562021688024/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:18 tscc-gpu-7-7 kernel: Lustre: 2539:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386798/real 1408386798]  req@ffff88045abdd800 x1476536280061424/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:18 tscc-1-61 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386798/real 1408386798]  req@ffff881031fe7400 x1476536929576532/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:18 tscc-2-13 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386798/real 1408386798]  req@ffff881ccbb15000 x1476519055846536/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:19 tscc-2-52 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408386799/real 1408386799]  req@ffff8820297ff800 x1476290267161232/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386810 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-0-31 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386794/real 1408386794]  req@ffff880812e8b800 x1476612857990744/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-0-28 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386794/real 1408386794]  req@ffff88100a434800 x1476547206508708/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-gpu-5-4 kernel: Lustre: 2626:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386760/real 1408386760]  req@ffff880805b33c00 x1476612337978376/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-gpu-5-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:33:26 tscc-1-48 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386759/real 1408386759]  req@ffff880cfd20bc00 x1476537001895280/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-1-48 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:33:26 tscc-1-40 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386759/real 1408386759]  req@ffff880f62dde400 x1476537020781972/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-1-40 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:33:26 tscc-0-10 kernel: Lustre: 2760:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386759/real 1408386759]  req@ffff880826789000 x1476547227469056/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-0-10 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:33:26 tscc-1-6 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386759/real 1408386759]  req@ffff880783006c00 x1476519511723760/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-1-6 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:33:26 tscc-3-43 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386759/real 1408386759]  req@ffff88050243fc00 x1476519948897280/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-3-43 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:33:26 tscc-0-36 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386759/real 1408386759]  req@ffff880826091800 x1476547204790168/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-0-36 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:33:26 tscc-gpu-3-1 kernel: Lustre: 2906:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386794/real 1408386794]  req@ffff880872f19000 x1476798077493388/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-3-48 kernel: Lustre: 2760:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386759/real 1408386759]  req@ffff881bec497000 x1476287317275512/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-3-48 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:33:26 tscc-3-51 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386759/real 1408386759]  req@ffff881008ab8400 x1476519881805056/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-3-51 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:33:26 tscc-0-2 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386759/real 1408386759]  req@ffff880832d8b000 x1476547435098952/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386806 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:26 tscc-0-2 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:33:35 tscc-1-48 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386806/real 1408386806]  req@ffff881cc45e9400 x1476537001895808/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:35 tscc-1-40 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386806/real 1408386806]  req@ffff880f2309f400 x1476537020782500/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:35 tscc-0-10 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386806/real 1408386806]  req@ffff880826789000 x1476547227469584/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:35 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386806/real 1408386806]  req@ffff880a609ba800 x1476519511724288/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:35 tscc-3-43 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386806/real 1408386806]  req@ffff88100495b000 x1476519948897808/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:35 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386806/real 1408386806]  req@ffff880826091000 x1476547204790696/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:35 tscc-3-48 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386806/real 1408386806]  req@ffff8818bf18a400 x1476287317276040/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:35 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386806/real 1408386806]  req@ffff881008ab8400 x1476519881805584/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:35 tscc-0-2 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386806/real 1408386806]  req@ffff8808162a8400 x1476547435099480/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386815 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:45 tscc-2-10 kernel: Lustre: 2969:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386777/real 1408386777]  req@ffff880ffb1bb400 x1476519079961516/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386825 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:45 tscc-2-10 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:33:45 tscc-gpu-6-3 kernel: Lustre: 2947:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386779/real 1408386779]  req@ffff88047084e000 x1476800383288772/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408386825 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:45 tscc-gpu-6-3 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:33:53 tscc-gpu-5-4 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880877501800: Communicating with 172.25.32.53@tcp, operation mds_sync failed with -107.
Aug 18 11:33:53 tscc-0-20 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-0-7 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-gpu-3-1 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-3-40 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-1-45 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-2-13 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-1-35 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-0-31 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-1-53 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-1-61 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-1-1 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-0-28 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-gpu-7-7 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-1-56 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-2-10 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-3-56 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-1-32 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-gpu-6-3 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-0-62 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-login1 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-2-52 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-0-15 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-1-11 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-2-39 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-0-57 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:53 tscc-1-14 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:33:59 tscc-gpu-5-4 kernel: Lustre: 2620:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386831/real 1408386831]  req@ffff880805b33000 x1476612337978880/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386839 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:33:59 tscc-gpu-5-4 kernel: Lustre: 2620:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:34:00 tscc-1-48 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386831/real 1408386831]  req@ffff881cc45e9000 x1476537001895812/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386840 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:34:00 tscc-1-40 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386831/real 1408386831]  req@ffff880ef1f8a800 x1476537020782504/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386840 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:34:00 tscc-0-10 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386831/real 1408386831]  req@ffff880826789000 x1476547227469588/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386840 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:34:00 tscc-1-6 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386831/real 1408386831]  req@ffff880783006000 x1476519511724292/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386840 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:34:00 tscc-3-43 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386831/real 1408386831]  req@ffff880713637000 x1476519948897812/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386840 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:34:00 tscc-0-36 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386831/real 1408386831]  req@ffff880826091800 x1476547204790700/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386840 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:34:00 tscc-3-48 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386831/real 1408386831]  req@ffff881bec497000 x1476287317276044/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386840 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:34:00 tscc-3-51 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386831/real 1408386831]  req@ffff881008ab8400 x1476519881805588/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386840 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:34:00 tscc-0-2 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386831/real 1408386831]  req@ffff880814c21c00 x1476547435099484/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408386840 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:34:16 tscc-gpu-5-4 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:34:16 tscc-1-48 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:34:16 tscc-1-40 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:34:16 tscc-0-10 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:34:16 tscc-1-6 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:34:16 tscc-0-36 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:34:16 tscc-3-43 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:34:16 tscc-3-48 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:34:16 tscc-3-51 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:34:16 tscc-0-2 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:34:45 tscc-gpu-5-4 kernel: Lustre: 2620:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408386831/real 1408386831]  req@ffff8808781db000 x1476612337978884/t0(0) o38->dolphin-MDT0000-mdc-ffff880877501800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408386885 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:35:06 tscc-gpu-5-4 kernel: Lustre: Evicted from dolphin-MDT0000_UUID (at 172.25.32.25@tcp) after server handle changed from 0x30845f5538e0b6e6 to 0x30845f5538ecb37f
Aug 18 11:35:06 tscc-gpu-5-4 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880877501800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:35:06 tscc-gpu-5-4 kernel: LustreError: 30974:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:35:06 tscc-gpu-5-4 kernel: LustreError: 27205:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x20000518d:0x5a:0x0] error -5.
Aug 18 11:35:06 tscc-gpu-5-4 kernel: LustreError: 31122:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff880877501800: Can't allocate new meta-sequence,rc -108
Aug 18 11:35:06 tscc-gpu-5-4 kernel: LustreError: 31122:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff880877501800: Can't allocate new sequence, rc -108
Aug 18 11:35:06 tscc-gpu-5-4 kernel: LustreError: 31122:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 11:35:06 tscc-gpu-5-4 kernel: LustreError: 30974:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 138 previous similar messages
Aug 18 11:35:06 tscc-gpu-5-4 kernel: LustreError: 29477:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID [0x14e00001:0xa0252bbc:0x0] error: rc = -108
Aug 18 11:35:06 tscc-gpu-5-4 kernel: LustreError: 29477:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 1 previous similar message
Aug 18 11:35:06 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:37:08 tscc-0-58 kernel: Lustre: Unmounted dolphin-client
Aug 18 11:37:37 tscc-gpu-6-0 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 11:37:42 tscc-gpu-6-0 kernel: Lustre: 2936:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387057/real 1408387057]  req@ffff880477a3ec00 x1476800866680836/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387062 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:37:43 tscc-gpu-6-0 kernel: LustreError: 2872:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880477a3e800 x1476800866680840/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 11:38:07 tscc-gpu-6-0 kernel: Lustre: 2936:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387082/real 1408387082]  req@ffff880477a3e800 x1476800866680928/t0(0) o38->dolphin-MDT0000-mdc-ffff88047230c800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408387087 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:38:27 tscc-gpu-6-0 kernel: Lustre: Layout lock feature supported.
Aug 18 11:38:27 tscc-gpu-6-0 kernel: Lustre: Mounted dolphin-client
Aug 18 11:40:05 tscc-gpu-6-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476c19c00: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 11:40:05 tscc-gpu-6-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476c19c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:40:05 tscc-gpu-6-1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880476c19c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:40:05 tscc-gpu-6-1 kernel: LustreError: 31345:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:40:05 tscc-gpu-6-1 kernel: LustreError: 31345:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 879 previous similar messages
Aug 18 11:40:05 tscc-gpu-6-1 kernel: LustreError: 31345:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x20000518b:0x24c:0x0] error -5.
Aug 18 11:40:05 tscc-gpu-6-1 kernel: LustreError: 31345:0:(vvp_io.c:1229:vvp_io_init()) Skipped 852 previous similar messages
Aug 18 11:40:05 tscc-gpu-6-1 kernel: LustreError: 5243:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID [0x20000518b:0x2ab:0x0] error: rc = -108
Aug 18 11:40:05 tscc-gpu-6-1 kernel: LustreError: 5243:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 3 previous similar messages
Aug 18 11:40:05 tscc-gpu-6-6 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880878d95000: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 11:40:05 tscc-gpu-6-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff880878d95000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:40:05 tscc-gpu-6-6 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880878d95000: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:40:05 tscc-gpu-6-6 kernel: LustreError: 15190:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:40:05 tscc-gpu-6-6 kernel: LustreError: 15190:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 19 previous similar messages
Aug 18 11:40:05 tscc-gpu-6-6 kernel: LustreError: 15190:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005126:0x451:0x0] error -5.
Aug 18 11:40:05 tscc-gpu-6-6 kernel: LustreError: 15190:0:(vvp_io.c:1229:vvp_io_init()) Skipped 12 previous similar messages
Aug 18 11:40:05 tscc-gpu-6-6 kernel: LustreError: 18485:0:(file.c:171:ll_close_inode_openhandle()) inode 144115536605741146 mdc close failed: rc = -108
Aug 18 11:40:05 tscc-gpu-6-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476c19c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:40:05 tscc-gpu-6-1 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:40:05 tscc-gpu-6-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff880878d95000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:41:00 tscc-1-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026059c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:41:00 tscc-gpu-5-4 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880877501800: Communicating with 172.25.32.25@tcp, operation mds_sync failed with -107.
Aug 18 11:41:00 tscc-gpu-5-4 kernel: LustreError: Skipped 5 previous similar messages
Aug 18 11:41:00 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:41:00 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 11:41:00 tscc-login1 kernel: LustreError: Skipped 1 previous similar message
Aug 18 11:41:00 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:41:00 tscc-gpu-5-4 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880877501800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:41:00 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:41:00 tscc-login1 kernel: LustreError: 56313:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:41:00 tscc-login1 kernel: LustreError: 56313:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 30 previous similar messages
Aug 18 11:41:00 tscc-gpu-5-4 kernel: LustreError: 30459:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:41:00 tscc-gpu-5-4 kernel: LustreError: 30459:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 373 previous similar messages
Aug 18 11:41:00 tscc-login1 kernel: LustreError: 56313:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff8810311a7800: Can't allocate new meta-sequence,rc -108
Aug 18 11:41:00 tscc-login1 kernel: LustreError: 56313:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff8810311a7800: Can't allocate new sequence, rc -108
Aug 18 11:41:00 tscc-login1 kernel: LustreError: 56313:0:(mdc_reint.c:235:mdc_create()) Can't alloc new fid, rc -108
Aug 18 11:41:00 tscc-login1 kernel: LustreError: 64399:0:(file.c:171:ll_close_inode_openhandle()) inode 350224747 mdc close failed: rc = -108
Aug 18 11:41:00 tscc-login1 kernel: LustreError: 64399:0:(file.c:171:ll_close_inode_openhandle()) Skipped 1 previous similar message
Aug 18 11:41:00 tscc-gpu-5-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff880877501800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:41:00 tscc-login1 kernel: LustreError: 56313:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005186:0x15:0x0] error -108.
Aug 18 11:41:00 tscc-login1 kernel: LustreError: 56313:0:(vvp_io.c:1229:vvp_io_init()) Skipped 24 previous similar messages
Aug 18 11:41:00 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:41:00 tscc-1-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882026059c00: Communicating with 172.25.32.25@tcp, operation mds_statfs failed with -107.
Aug 18 11:41:00 tscc-1-1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff882026059c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:41:00 tscc-1-1 kernel: LustreError: 57080:0:(llite_lib.c:1610:ll_statfs_internal()) md_statfs fails: rc = -5
Aug 18 11:41:00 tscc-1-1 kernel: LustreError: 57080:0:(llite_lib.c:1610:ll_statfs_internal()) Skipped 1 previous similar message
Aug 18 11:41:00 tscc-1-1 kernel: LustreError: 57080:0:(file.c:171:ll_close_inode_openhandle()) inode 144115537545265268 mdc close failed: rc = -108
Aug 18 11:41:00 tscc-1-1 kernel: LustreError: 57080:0:(file.c:171:ll_close_inode_openhandle()) Skipped 1 previous similar message
Aug 18 11:41:00 tscc-1-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026059c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:43:56 tscc-1-12 kernel: Lustre: 2765:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387426/real 1408387426]  req@ffff88116d719400 x1476547036066736/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-1-12 kernel: Lustre: 2765:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-1-12 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-gpu-3-4 kernel: Lustre: 2558:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387424/real 1408387424]  req@ffff881064bee400 x1476571902041128/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-gpu-3-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-0-63 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387419/real 1408387419]  req@ffff881032f47800 x1476798012488800/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-0-63 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-0-60 kernel: Lustre: 2818:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387427/real 1408387427]  req@ffff881030d36800 x1476700762685308/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-0-60 kernel: Lustre: 2818:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-0-60 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-gpu-5-2 kernel: Lustre: 2632:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387428/real 1408387428]  req@ffff8808601d3400 x1476558499557288/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-gpu-5-2 kernel: Lustre: 2632:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Aug 18 11:43:56 tscc-gpu-5-2 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-2-55 kernel: Lustre: 2965:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387423/real 1408387423]  req@ffff88063a5e1400 x1476519002427908/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-2-55 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-pdafm-7-0 kernel: Lustre: 3155:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387424/real 1408387424]  req@ffff88100b312800 x1476534244802852/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-pdafm-7-0 kernel: Lustre: 3155:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-pdafm-7-0 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-0-5 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387425/real 1408387425]  req@ffff880816395c00 x1476638712329552/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-0-5 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-0-5 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-3-54 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387422/real 1408387422]  req@ffff880bfdab4c00 x1476290785586764/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-3-54 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-0-18 kernel: Lustre: 2769:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387424/real 1408387424]  req@ffff880823036000 x1476547249485528/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-0-18 kernel: Lustre: 2769:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-0-18 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-1-46 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387428/real 1408387428]  req@ffff8810156e5400 x1476536973605608/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-1-46 kernel: Lustre: 2724:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-1-46 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-0-0 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387423/real 1408387423]  req@ffff88083007d800 x1476519380610940/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-0-0 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-0-0 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-0-39 kernel: Lustre: 2222:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387423/real 1408387423]  req@ffff8805952ddc00 x1475608471422580/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-0-39 kernel: Lustre: 2222:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-0-39 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-3-41 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387419/real 1408387419]  req@ffff8802a8677c00 x1476519856771200/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-3-41 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-3-41 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-1-41 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387428/real 1408387428]  req@ffff880d0bf95800 x1476537045927252/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-1-41 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-1-41 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-gpu-5-7 kernel: Lustre: 2554:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387423/real 1408387423]  req@ffff88043df1a000 x1476612406276800/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-gpu-5-7 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-1-4 kernel: Lustre: 2736:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387429/real 1408387429]  req@ffff880efed46800 x1475918802758776/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-1-4 kernel: Lustre: 2736:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:43:56 tscc-1-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-3-46 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387427/real 1408387427]  req@ffff881024fa7c00 x1476519914787492/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-3-46 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 11:43:56 tscc-3-46 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-0-13 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387424/real 1408387424]  req@ffff880429a04800 x1476634825401212/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-0-13 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-0-13 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-3-49 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387424/real 1408387424]  req@ffff880d8451f400 x1476519923746764/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-3-49 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-3-49 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-1-33 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387427/real 1408387427]  req@ffff880eb7d40c00 x1476536972534648/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-1-33 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-1-33 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-gpu-3-2 kernel: Lustre: 2909:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387427/real 1408387427]  req@ffff88085deabc00 x1476798148802360/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-gpu-3-2 kernel: Lustre: 2909:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-gpu-3-2 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-2-37 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387424/real 1408387424]  req@ffff880d7b8a4c00 x1476570224078144/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-2-37 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-2-37 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-0-21 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387423/real 1408387423]  req@ffff880820b5e400 x1476699991542464/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-0-21 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-0-21 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:43:56 tscc-1-59 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387428/real 1408387428]  req@ffff88053f7d7c00 x1476536863850688/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:43:56 tscc-1-59 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:43:56 tscc-1-59 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:44:02 tscc-1-4 kernel: Lustre: 2732:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff8810266d8c00 x1475918802758800/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387442 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:03 tscc-gpu-5-2 kernel: Lustre: 2625:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff88082982c000 x1476558499557536/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387443 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:03 tscc-1-46 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff8810156e5400 x1476536973605872/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387443 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:03 tscc-1-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff880edfae9800 x1476537045927516/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387443 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:03 tscc-1-59 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff88064e5c1000 x1476536863850952/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387443 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:04 tscc-0-60 kernel: Lustre: 2813:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff881030583800 x1476700762685572/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387444 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:04 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff880ee31d7400 x1476519914787640/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387444 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:04 tscc-1-33 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff88102529c800 x1476536972534912/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387444 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:04 tscc-gpu-3-2 kernel: Lustre: 2905:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff88085dce2c00 x1476798148802624/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387444 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:05 tscc-1-12 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff881ae91a3400 x1476547036067000/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387445 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:06 tscc-0-5 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff8808317bac00 x1476638712329816/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387446 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:07 tscc-gpu-3-4 kernel: Lustre: 2555:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff881072424800 x1476571902041392/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387447 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:07 tscc-pdafm-7-0 kernel: Lustre: 3148:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff881010b31000 x1476534244803116/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387447 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:07 tscc-0-18 kernel: Lustre: 2753:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff88082243a800 x1476547249485792/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387447 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:07 tscc-0-13 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff8804214eec00 x1476634825401476/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387447 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:07 tscc-3-49 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff880506268400 x1476519923747028/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387447 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:07 tscc-2-37 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff880fb8e01c00 x1476570224078408/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387447 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:08 tscc-2-55 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff8804be527400 x1476519002428172/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387448 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:08 tscc-0-0 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff8808145d9400 x1476519380611204/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387448 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:08 tscc-0-39 kernel: Lustre: 2216:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff88081ce91000 x1475608471422844/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387448 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:08 tscc-gpu-5-7 kernel: Lustre: 2551:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff880458711400 x1476612406277060/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387448 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:08 tscc-0-21 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff880831d95800 x1476699991542728/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387448 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:09 tscc-3-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff880bfdab4000 x1476290785587516/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387449 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:12 tscc-0-63 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff88103272f800 x1476798012489064/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387452 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:12 tscc-3-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387436/real 1408387436]  req@ffff880bd6d95400 x1476519856771344/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387452 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:21 tscc-gpu-6-6 kernel: Lustre: 2532:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387432/real 1408387432]  req@ffff880855d02800 x1476558758288408/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387461 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:21 tscc-gpu-6-6 kernel: Lustre: 2532:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:44:21 tscc-gpu-6-6 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:44:21 tscc-1-9 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387441/real 1408387441]  req@ffff88116f629800 x1476519677626680/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387461 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:21 tscc-1-9 kernel: Lustre: 2752:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:44:21 tscc-1-9 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:44:21 tscc-2-50 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387432/real 1408387432]  req@ffff881fc1211c00 x1476518982748044/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387461 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:21 tscc-2-50 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:44:21 tscc-2-50 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:44:21 tscc-0-34 kernel: Lustre: 2772:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387432/real 1408387432]  req@ffff880796f93800 x1476614108125420/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408387461 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:21 tscc-0-34 kernel: Lustre: 2772:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:44:21 tscc-0-34 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:44:21 tscc-gpu-6-6 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880878d95000: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 11:44:21 tscc-gpu-6-6 kernel: LustreError: Skipped 3 previous similar messages
Aug 18 11:44:21 tscc-gpu-6-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff880878d95000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:44:21 tscc-gpu-5-2 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88086f941c00: Communicating with 172.25.32.25@tcp, operation mds_sync failed with -107.
Aug 18 11:44:21 tscc-gpu-5-2 kernel: LustreError: Skipped 28 previous similar messages
Aug 18 11:44:21 tscc-gpu-5-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff88086f941c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:44:21 tscc-gpu-6-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476c19c00: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 11:44:21 tscc-gpu-6-1 kernel: LustreError: Skipped 9 previous similar messages
Aug 18 11:44:21 tscc-gpu-6-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476c19c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:44:21 tscc-gpu-5-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876ea6c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 11:44:21 tscc-gpu-5-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876ea6c00: Communicating with 172.25.32.25@tcp, operation mds_close failed with -107.
Aug 18 11:44:21 tscc-gpu-5-7 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 11:44:21 tscc-0-34 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:21 tscc-gpu-5-7 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880876ea6c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:44:21 tscc-2-50 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:21 tscc-gpu-6-6 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:21 tscc-gpu-6-6 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880878d95000: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:44:21 tscc-gpu-6-6 kernel: LustreError: 15190:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:44:21 tscc-gpu-6-6 kernel: LustreError: 15190:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 105 previous similar messages
Aug 18 11:44:21 tscc-gpu-6-1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880476c19c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:44:21 tscc-gpu-6-6 kernel: LustreError: 15190:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005193:0x288:0x0] error -5.
Aug 18 11:44:21 tscc-gpu-6-6 kernel: LustreError: 15190:0:(vvp_io.c:1229:vvp_io_init()) Skipped 31 previous similar messages
Aug 18 11:44:21 tscc-gpu-5-7 kernel: LustreError: 30854:0:(file.c:171:ll_close_inode_openhandle()) inode 144115538317017099 mdc close failed: rc = -5
Aug 18 11:44:21 tscc-gpu-5-7 kernel: LustreError: 30854:0:(file.c:171:ll_close_inode_openhandle()) Skipped 13 previous similar messages
Aug 18 11:44:21 tscc-gpu-5-2 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff88086f941c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 11:44:21 tscc-gpu-6-1 kernel: LustreError: 31345:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:44:21 tscc-gpu-6-1 kernel: LustreError: 31345:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 44 previous similar messages
Aug 18 11:44:21 tscc-gpu-6-1 kernel: LustreError: 31345:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005194:0x311:0x0] error -5.
Aug 18 11:44:21 tscc-gpu-6-1 kernel: LustreError: 3636:0:(file.c:171:ll_close_inode_openhandle()) inode 144115538451235589 mdc close failed: rc = -108
Aug 18 11:44:21 tscc-gpu-6-1 kernel: LustreError: 3636:0:(file.c:171:ll_close_inode_openhandle()) Skipped 10 previous similar messages
Aug 18 11:44:21 tscc-gpu-6-1 kernel: LustreError: 31345:0:(vvp_io.c:1229:vvp_io_init()) Skipped 15 previous similar messages
Aug 18 11:44:21 tscc-gpu-5-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876ea6c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:44:21 tscc-gpu-5-2 kernel: LustreError: 28087:0:(file.c:171:ll_close_inode_openhandle()) inode 144115538367348931 mdc close failed: rc = -5
Aug 18 11:44:21 tscc-gpu-5-2 kernel: LustreError: 2179:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 11:44:21 tscc-gpu-5-2 kernel: LustreError: 2179:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 1873 previous similar messages
Aug 18 11:44:21 tscc-gpu-5-2 kernel: LustreError: 28087:0:(file.c:171:ll_close_inode_openhandle()) Skipped 13 previous similar messages
Aug 18 11:44:21 tscc-gpu-5-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff88086f941c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:44:21 tscc-gpu-5-2 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:44:21 tscc-gpu-6-6 kernel: LustreError: 20407:0:(ldlm_resource.c:809:ldlm_resource_complain()) dolphin-MDT0000-mdc-ffff880878d95000: namespace resource [0x200005193:0x2be:0x0].0 (ffff88084620ce40) refcount nonzero (1) after lock cleanup; forcing cleanup.
Aug 18 11:44:21 tscc-gpu-6-6 kernel: LustreError: 20407:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x200005193:0x2be:0x0].0 (ffff88084620ce40) refcount = 2
Aug 18 11:44:21 tscc-1-9 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:21 tscc-gpu-6-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476c19c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 11:44:27 tscc-1-4 kernel: Lustre: 2732:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff881028741800 x1475918802758824/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387467 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:28 tscc-gpu-5-2 kernel: Lustre: 2625:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff880857137400 x1476558499557684/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387468 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:28 tscc-1-46 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff88101e7c9c00 x1476536973605876/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387468 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:28 tscc-1-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff880d0bf95c00 x1476537045927520/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387468 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:28 tscc-1-59 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff88081be93400 x1476536863850960/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387468 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:29 tscc-0-60 kernel: Lustre: 2813:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff8810320cf800 x1476700762685576/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387469 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:29 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff880ee31d7400 x1476519914787820/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387469 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:29 tscc-1-33 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff880dace59400 x1476536972534916/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387469 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:29 tscc-gpu-3-2 kernel: Lustre: 2905:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff88085deab800 x1476798148802628/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387469 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:30 tscc-1-12 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff8814c474dc00 x1476547036067004/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387470 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:31 tscc-0-5 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff880816395000 x1476638712329820/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387471 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:32 tscc-gpu-3-4 kernel: Lustre: 2555:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff88105784e800 x1476571902041396/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387472 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:32 tscc-pdafm-7-0 kernel: Lustre: 3148:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff881010b31000 x1476534244803120/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387472 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:32 tscc-0-18 kernel: Lustre: 2753:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff880823036000 x1476547249485796/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387472 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:32 tscc-0-13 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff8804214ee400 x1476634825401480/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387472 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:32 tscc-3-49 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff880d8451fc00 x1476519923747032/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387472 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:32 tscc-2-37 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff881030af7c00 x1476570224078412/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387472 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:33 tscc-2-55 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff881068391000 x1476519002428184/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387473 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:33 tscc-0-0 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff88083007dc00 x1476519380611208/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387473 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:33 tscc-0-39 kernel: Lustre: 2216:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff88073ccbac00 x1475608471422848/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387473 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:33 tscc-gpu-5-7 kernel: Lustre: 2551:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff88047785a000 x1476612406277104/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387473 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:33 tscc-0-21 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff880820b5e800 x1476699991542732/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387473 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:34 tscc-3-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff880cd8c7a000 x1476290785588256/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387474 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:37 tscc-0-63 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff881032f47400 x1476798012489068/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:37 tscc-3-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408387461/real 1408387461]  req@ffff881008a70000 x1476519856771596/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408387477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:44:46 tscc-1-12 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-gpu-3-4 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-0-63 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-0-60 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-gpu-5-2 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-2-55 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-pdafm-7-0 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-0-5 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-3-54 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-0-18 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-1-46 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-0-0 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-0-39 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-1-41 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-3-41 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-3-41 kernel: Lustre: Skipped 1 previous similar message
Aug 18 11:44:46 tscc-1-59 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-gpu-5-7 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-1-4 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-3-46 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-0-13 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-1-33 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-3-49 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-gpu-3-2 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-2-37 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:44:46 tscc-0-21 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:53:58 tscc-gpu-3-1 kernel: Lustre: Unmounted dolphin-client
Aug 18 11:53:59 tscc-1-38 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388032/real 1408388032]  req@ffff882026632400 x1476575145596504/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388039 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:53:59 tscc-1-38 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:53:59 tscc-0-26 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388032/real 1408388032]  req@ffff880fc0989400 x1476612864128856/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388039 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:53:59 tscc-0-26 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:53:59 tscc-0-26 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:53:59 tscc-0-29 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388032/real 1408388032]  req@ffff880195f54800 x1476612872593424/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388039 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:53:59 tscc-0-29 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 11:53:59 tscc-0-29 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 11:53:59 tscc-0-29 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:53:59 tscc-0-26 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:53:59 tscc-1-38 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 11:55:08 tscc-gpu-3-2 kernel: Lustre: Unmounted dolphin-client
Aug 18 11:56:25 tscc-gpu-3-3 kernel: Lustre: Unmounted dolphin-client
Aug 18 11:57:37 tscc-gpu-3-1 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 11:57:39 tscc-gpu-6-3 kernel: Lustre: Unmounted dolphin-client
Aug 18 11:57:42 tscc-gpu-3-1 kernel: Lustre: 2492:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388257/real 1408388257]  req@ffff88086fa02c00 x1476802124972036/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388262 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:57:43 tscc-gpu-3-1 kernel: LustreError: 2428:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88086fa02800 x1476802124972040/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 11:57:53 tscc-gpu-3-1 kernel: LustreError: 2522:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88086fa02000 x1476802124972048/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 11:58:07 tscc-gpu-3-1 kernel: Lustre: 2492:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388282/real 1408388282]  req@ffff88086fa02800 x1476802124972132/t0(0) o38->dolphin-MDT0000-mdc-ffff880876650000@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408388287 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:58:14 tscc-gpu-3-3 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 11:58:19 tscc-gpu-3-3 kernel: Lustre: 2503:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388294/real 1408388294]  req@ffff881070f0ec00 x1476802163769348/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388299 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:58:20 tscc-gpu-3-3 kernel: LustreError: 2439:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff881070f0e800 x1476802163769352/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 11:58:27 tscc-gpu-3-1 kernel: Lustre: Layout lock feature supported.
Aug 18 11:58:27 tscc-gpu-3-1 kernel: Lustre: Mounted dolphin-client
Aug 18 11:58:28 tscc-gpu-3-3 kernel: LustreError: 2533:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff881070f0ec00 x1476802163769360/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 11:58:29 tscc-gpu-3-2 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 11:58:34 tscc-gpu-3-2 kernel: Lustre: 2494:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388309/real 1408388309]  req@ffff8804755a6c00 x1476802179497988/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388314 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:58:35 tscc-gpu-3-2 kernel: LustreError: 2430:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff8804755a6800 x1476802179497992/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 11:58:44 tscc-gpu-3-3 kernel: Lustre: 2503:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388319/real 1408388319]  req@ffff880870c3ac00 x1476802163769444/t0(0) o38->dolphin-MDT0000-mdc-ffff881073d8e800@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408388324 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:58:47 tscc-pdafm-7-2 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 11:58:52 tscc-pdafm-7-2 kernel: Lustre: 3151:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388327/real 1408388327]  req@ffff885010af0c00 x1476802198372356/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388332 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:58:53 tscc-pdafm-7-2 kernel: LustreError: 3066:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff885010af0800 x1476802198372360/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 11:58:59 tscc-gpu-3-2 kernel: Lustre: 2494:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388334/real 1408388334]  req@ffff8804755a6800 x1476802179498080/t0(0) o38->dolphin-MDT0000-mdc-ffff880476f45c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408388339 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:59:02 tscc-pdafm-7-2 kernel: LustreError: 3201:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff8810108a0c00 x1476802198372368/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 11:59:04 tscc-gpu-3-3 kernel: Lustre: Layout lock feature supported.
Aug 18 11:59:05 tscc-gpu-3-3 kernel: Lustre: Mounted dolphin-client
Aug 18 11:59:17 tscc-pdafm-7-2 kernel: Lustre: 3151:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388352/real 1408388352]  req@ffff885010af0800 x1476802198372452/t0(0) o38->dolphin-MDT0000-mdc-ffff88501140c400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408388357 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:59:19 tscc-gpu-3-2 kernel: Lustre: Layout lock feature supported.
Aug 18 11:59:19 tscc-gpu-3-2 kernel: Lustre: Mounted dolphin-client
Aug 18 11:59:37 tscc-pdafm-7-2 kernel: Lustre: Layout lock feature supported.
Aug 18 11:59:37 tscc-pdafm-7-2 kernel: Lustre: Mounted dolphin-client
Aug 18 11:59:45 tscc-gpu-6-3 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 11:59:50 tscc-gpu-6-3 kernel: Lustre: 2605:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388385/real 1408388385]  req@ffff8804624cbc00 x1476802259189764/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388390 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 11:59:51 tscc-gpu-6-3 kernel: LustreError: 2541:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff8804624cb800 x1476802259189768/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 11:59:57 tscc-gpu-6-3 kernel: LustreError: 2635:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880876134c00 x1476802259189776/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 12:00:16 tscc-gpu-6-3 kernel: Lustre: 2605:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388411/real 1408388411]  req@ffff8804624cb800 x1476802259189860/t0(0) o38->dolphin-MDT0000-mdc-ffff880476039c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408388416 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:00:36 tscc-gpu-6-3 kernel: Lustre: Layout lock feature supported.
Aug 18 12:00:36 tscc-gpu-6-3 kernel: Lustre: Mounted dolphin-client
Aug 18 12:09:00 tscc-1-43 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388933/real 1408388933]  req@ffff880d71422000 x1476570186913300/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388940 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:00 tscc-1-43 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 12:09:00 tscc-1-43 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:02 tscc-gpu-3-2 kernel: Lustre: 2495:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388935/real 1408388935]  req@ffff8808764c3c00 x1476802179504172/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388942 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:02 tscc-gpu-3-2 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:02 tscc-gpu-6-1 kernel: Lustre: 2525:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388935/real 1408388935]  req@ffff88075bd7c800 x1476558706975896/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388942 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:02 tscc-gpu-6-1 kernel: Lustre: 2525:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 12:09:02 tscc-gpu-6-1 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-gpu-6-6 kernel: Lustre: 2530:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff88086d32a800 x1476558758407736/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-gpu-6-6 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-1-12 kernel: Lustre: 2762:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff884010799800 x1476547036098684/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-1-12 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-gpu-3-4 kernel: Lustre: 2557:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff881063149c00 x1476571902057004/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-gpu-3-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-0-63 kernel: Lustre: 2738:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff880fd98d4c00 x1476798012504676/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-0-63 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-0-60 kernel: Lustre: 2818:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff881031717000 x1476700762703364/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-0-60 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-gpu-5-2 kernel: Lustre: 2628:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff8808772c0400 x1476558499626496/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-gpu-5-2 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-2-55 kernel: Lustre: 2972:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff881066287c00 x1476519002447824/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-2-55 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-pdafm-7-0 kernel: Lustre: 3174:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff887011d0a400 x1476534244818728/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-pdafm-7-0 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-1-9 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff88116f629800 x1476519677642292/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-1-9 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-0-34 kernel: Lustre: 2765:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff880796f93800 x1476614108141296/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-0-34 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-2-50 kernel: Lustre: 2957:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff881fa05bf000 x1476518982763920/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-2-50 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-0-5 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff88082912cc00 x1476638712345428/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-0-5 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-3-54 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff880ada6c3800 x1476290785613436/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-3-54 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-0-18 kernel: Lustre: 2766:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff880833026400 x1476547249501404/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-0-18 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-1-46 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff880f0c4cd000 x1476536973621484/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-1-46 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-0-0 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff88083007d000 x1476519380626816/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-0-0 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-0-39 kernel: Lustre: 2218:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff8806d2a57c00 x1475608471438448/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-0-39 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-1-41 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff882009886000 x1476537045943128/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-1-41 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-3-41 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff880af7a2b800 x1476519856870376/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-3-41 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-1-59 kernel: Lustre: 2738:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff88053f7d7000 x1476536863867020/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-1-59 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-gpu-5-7 kernel: Lustre: 2562:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff8803bb97d400 x1476612406319616/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-gpu-5-7 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-1-4 kernel: Lustre: 2742:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff88101eb77800 x1475918802837448/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-1-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-0-13 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff8804214eec00 x1476634825417088/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-0-13 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-3-46 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff881fc8870400 x1476519914893092/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-3-46 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-3-49 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff880304151c00 x1476519923762640/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-3-49 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-1-33 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff880ca8254400 x1476536972550520/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-1-33 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-2-37 kernel: Lustre: 2723:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff88101579c400 x1476570224094016/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-2-37 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:03 tscc-0-21 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff880831e74400 x1476699991558340/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:03 tscc-0-21 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:04 tscc-1-54 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388937/real 1408388937]  req@ffff880dc020c400 x1476536988283900/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388944 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:04 tscc-1-54 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:06 tscc-1-38 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388939/real 1408388939]  req@ffff881e17743400 x1476575145610292/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388946 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:06 tscc-1-38 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:06 tscc-1-43 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388940/real 1408388940]  req@ffff880d0bd94c00 x1476570186913564/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388946 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:06 tscc-0-26 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388939/real 1408388939]  req@ffff880f58055800 x1476612864138396/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388946 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:06 tscc-0-26 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:07 tscc-0-29 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388939/real 1408388939]  req@ffff88083098c400 x1476612872602964/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388947 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:07 tscc-0-29 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:08 tscc-gpu-3-2 kernel: Lustre: 2494:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388942/real 1408388942]  req@ffff8808763c7000 x1476802179504436/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388948 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:08 tscc-2-11 kernel: Lustre: 2960:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388941/real 1408388941]  req@ffff8820687b7400 x1476519079146548/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388948 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:08 tscc-2-11 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:08 tscc-gpu-6-1 kernel: Lustre: 2523:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388942/real 1408388942]  req@ffff88086e942400 x1476558706976184/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388948 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-gpu-6-6 kernel: Lustre: 2524:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff880848db0800 x1476558758408000/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-1-12 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff883ee7e1e000 x1476547036098688/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-gpu-3-4 kernel: Lustre: 2555:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff88106d642c00 x1476571902057268/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-0-63 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff880fd98d4c00 x1476798012504940/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-0-60 kernel: Lustre: 2813:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff881030c87000 x1476700762703628/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-gpu-5-2 kernel: Lustre: 2625:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff8807a4075000 x1476558499626760/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-2-55 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff8810573f9400 x1476519002448088/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-pdafm-7-0 kernel: Lustre: 3148:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff88700c839400 x1476534244818992/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-1-9 kernel: Lustre: 2738:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff8811adbe7000 x1476519677642556/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-0-34 kernel: Lustre: 2756:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff880816c53000 x1476614108141560/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-2-50 kernel: Lustre: 2956:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff881c3016f400 x1476518982764184/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-0-5 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff8808140de800 x1476638712345692/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-3-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff880cd41eb000 x1476290785613680/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-0-18 kernel: Lustre: 2753:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff88082013a400 x1476547249501668/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-1-46 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff880f0c4cd000 x1476536973621748/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-0-0 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff8808159b3c00 x1476519380627080/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-0-39 kernel: Lustre: 2216:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff88081de21400 x1475608471438712/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-1-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff882009886000 x1476537045943392/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-3-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff881008d3f800 x1476519856870560/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-1-59 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff88081bb7a800 x1476536863867284/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-gpu-5-7 kernel: Lustre: 2551:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff88045107d800 x1476612406319876/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-1-4 kernel: Lustre: 2732:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff880be9a04c00 x1475918802837460/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-0-13 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff88083378d800 x1476634825417352/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff881ff24e9800 x1476519914893184/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-3-49 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff880af6af7c00 x1476519923762904/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-1-33 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff880dace59800 x1476536972550784/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-2-37 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff880f9b76a000 x1476570224094280/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:09 tscc-0-21 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388943/real 1408388943]  req@ffff880810734400 x1476699991558604/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:12 tscc-1-38 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388946/real 1408388946]  req@ffff881e5dd57400 x1476575145610312/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388952 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:12 tscc-0-26 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388946/real 1408388946]  req@ffff8810044e3800 x1476612864138660/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388952 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:14 tscc-2-11 kernel: Lustre: 2959:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388948/real 1408388948]  req@ffff881ce0707c00 x1476519079146812/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388954 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:14 tscc-0-29 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388947/real 1408388947]  req@ffff880830be6000 x1476612872603228/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388954 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:20 tscc-pdafm-7-2 kernel: Lustre: 3179:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388952/real 1408388952]  req@ffff8810052e4c00 x1476802198378544/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388960 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:20 tscc-pdafm-7-2 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:24 tscc-1-51 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388957/real 1408388957]  req@ffff881defc8fc00 x1476542205956192/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388964 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:24 tscc-1-51 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:24 tscc-0-8 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388957/real 1408388957]  req@ffff8808139b8800 x1476547167692656/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408388964 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:24 tscc-0-8 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Aug 18 12:09:24 tscc-0-8 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:09:27 tscc-pdafm-7-2 kernel: Lustre: 3151:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388960/real 1408388960]  req@ffff880fe5982400 x1476802198378808/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388967 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:30 tscc-0-8 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388964/real 1408388964]  req@ffff8808118d4c00 x1476547167692920/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388970 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:31 tscc-1-43 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388965/real 1408388965]  req@ffff880cbc3dc400 x1476570186913568/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388971 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:33 tscc-gpu-3-2 kernel: Lustre: 2494:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388967/real 1408388967]  req@ffff88086a515800 x1476802179504440/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388973 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:33 tscc-gpu-6-1 kernel: Lustre: 2523:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388967/real 1408388967]  req@ffff88086cfd8000 x1476558706976300/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388973 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-gpu-6-6 kernel: Lustre: 2524:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff8805285a5c00 x1476558758408004/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-1-12 kernel: Lustre: 2755:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff88400f9f1400 x1476547036098724/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-gpu-3-4 kernel: Lustre: 2555:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff88105784e800 x1476571902057272/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-0-63 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff880fd98d4c00 x1476798012504944/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-0-60 kernel: Lustre: 2813:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff8810320cfc00 x1476700762703632/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-gpu-5-2 kernel: Lustre: 2625:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff88084e9bb000 x1476558499626920/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-2-55 kernel: Lustre: 2963:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff8810573f9800 x1476519002448092/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-pdafm-7-0 kernel: Lustre: 3148:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff88700c839400 x1476534244818996/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-1-9 kernel: Lustre: 2738:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff881ffe530000 x1476519677642560/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-0-34 kernel: Lustre: 2756:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff880796f93400 x1476614108141564/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-2-50 kernel: Lustre: 2956:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff881cec36cc00 x1476518982764188/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-0-5 kernel: Lustre: 2716:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff88082912c000 x1476638712345696/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-3-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff88053c92ec00 x1476290785613708/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-0-18 kernel: Lustre: 2753:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff880833026400 x1476547249501672/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-1-46 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff880d8cea5000 x1476536973621752/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-0-0 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff88083007d800 x1476519380627084/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-0-39 kernel: Lustre: 2216:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff8803a2348000 x1475608471438716/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-1-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff881ff3d4c400 x1476537045943396/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-3-41 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff881008bc8000 x1476519856870604/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-1-59 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff88081be93400 x1476536863867296/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-gpu-5-7 kernel: Lustre: 2551:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff88043dd51400 x1476612406319900/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-1-4 kernel: Lustre: 2732:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff880dc0193400 x1475918802837716/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-0-13 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff8807be144c00 x1476634825417356/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff881c3ed7c800 x1476519914893240/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-3-49 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff880304151c00 x1476519923762908/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-1-33 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff88102507f400 x1476536972550788/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-2-37 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff880d66dd8400 x1476570224094284/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:34 tscc-0-21 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388968/real 1408388968]  req@ffff880831e74000 x1476699991558608/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388974 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:35 tscc-1-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388969/real 1408388969]  req@ffff880e4a254000 x1476536988284168/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388975 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:35 tscc-1-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:09:37 tscc-1-38 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388971/real 1408388971]  req@ffff881ff8a3e400 x1476575145610332/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:37 tscc-0-26 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388971/real 1408388971]  req@ffff880f58055400 x1476612864138664/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:39 tscc-gpu-3-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476f45c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:39 tscc-2-11 kernel: Lustre: 2959:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388973/real 1408388973]  req@ffff8820687b7000 x1476519079146816/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388979 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:39 tscc-3-41 kernel: Lustre: 35662:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388933/real 1408388933]  req@ffff880cd4045400 x1476519856870372/t0(0) o101->dolphin-MDT0000-mdc-ffff882026020800@172.25.32.25@tcp:12/10 lens 616/1136 e 0 to 1 dl 1408388979 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1
Aug 18 12:09:39 tscc-3-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026020800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:39 tscc-0-29 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388972/real 1408388972]  req@ffff88083098cc00 x1476612872603232/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388979 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:40 tscc-gpu-6-6 kernel: Lustre: 2531:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff88086d32a000 x1476558758407740/t0(0) o400->dolphin-MDT0000-mdc-ffff880878d95000@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408388980 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:40 tscc-gpu-6-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff880878d95000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-gpu-3-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff88087229d800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-0-63 kernel: Lustre: dolphin-MDT0000-mdc-ffff881031346c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-0-60 kernel: Lustre: dolphin-MDT0000-mdc-ffff881030ca2400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-2-55 kernel: Lustre: dolphin-MDT0000-mdc-ffff882068dc4c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-pdafm-7-0 kernel: Lustre: dolphin-MDT0000-mdc-ffff885010079800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-1-9 kernel: Lustre: 2746:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff88116f629000 x1476519677642296/t0(0) o400->dolphin-MDT0000-mdc-ffff8840101b8c00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408388980 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:40 tscc-1-9 kernel: Lustre: dolphin-MDT0000-mdc-ffff8840101b8c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-0-34 kernel: Lustre: 2766:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff880796f93000 x1476614108141300/t0(0) o400->dolphin-MDT0000-mdc-ffff881030f1ac00@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408388980 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:40 tscc-0-34 kernel: Lustre: dolphin-MDT0000-mdc-ffff881030f1ac00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-3-54 kernel: Lustre: dolphin-MDT0000-mdc-ffff881026a4a400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-2-50 kernel: Lustre: 2958:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388936/real 1408388936]  req@ffff881ff7a48000 x1476518982763924/t0(0) o400->dolphin-MDT0000-mdc-ffff881065a37400@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408388980 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:40 tscc-2-50 kernel: Lustre: dolphin-MDT0000-mdc-ffff881065a37400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-0-5 kernel: Lustre: dolphin-MDT0000-mdc-ffff88103170a000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-0-18 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810343ce400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-1-46 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101d826800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-0-0 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810326bb000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-0-39 kernel: Lustre: dolphin-MDT0000-mdc-ffff88081c7a2000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-1-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026991400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-1-59 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831706400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-0-13 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830ad3800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-3-49 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026177c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-1-33 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810264d1400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-2-37 kernel: Lustre: dolphin-MDT0000-mdc-ffff881033c5a800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:40 tscc-0-21 kernel: Lustre: dolphin-MDT0000-mdc-ffff881032040400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:41 tscc-1-54 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025a03800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:42 tscc-gpu-6-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476c19c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:43 tscc-0-26 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388939/real 1408388939]  req@ffff880f58055000 x1476612864138400/t0(0) o400->dolphin-MDT0000-mdc-ffff8810314d8800@172.25.32.25@tcp:12/10 lens 224/224 e 0 to 1 dl 1408388983 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:43 tscc-0-26 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810314d8800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:44 tscc-gpu-5-2 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88086f941c00: Communicating with 172.25.32.25@tcp, operation obd_ping failed with -107.
Aug 18 12:09:44 tscc-gpu-5-2 kernel: LustreError: Skipped 6 previous similar messages
Aug 18 12:09:44 tscc-gpu-5-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff88086f941c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:44 tscc-gpu-5-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876ea6c00: Communicating with 172.25.32.25@tcp, operation obd_ping failed with -107.
Aug 18 12:09:44 tscc-gpu-5-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876ea6c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:09:44 tscc-0-63 kernel: Lustre: dolphin-MDT0000-mdc-ffff881031346c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-gpu-3-4 kernel: Lustre: dolphin-MDT0000-mdc-ffff88087229d800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-0-60 kernel: Lustre: dolphin-MDT0000-mdc-ffff881030ca2400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-gpu-3-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476f45c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-1-9 kernel: Lustre: dolphin-MDT0000-mdc-ffff8840101b8c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-0-0 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810326bb000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-gpu-6-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff880878d95000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-gpu-6-6 kernel: Lustre: Skipped 1 previous similar message
Aug 18 12:09:44 tscc-0-5 kernel: Lustre: dolphin-MDT0000-mdc-ffff88103170a000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-2-55 kernel: Lustre: dolphin-MDT0000-mdc-ffff882068dc4c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-1-59 kernel: Lustre: dolphin-MDT0000-mdc-ffff880831706400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-3-49 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026177c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-pdafm-7-0 kernel: Lustre: dolphin-MDT0000-mdc-ffff885010079800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-1-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026991400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-3-54 kernel: Lustre: dolphin-MDT0000-mdc-ffff881026a4a400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-0-39 kernel: Lustre: dolphin-MDT0000-mdc-ffff88081c7a2000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-0-13 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830ad3800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-2-50 kernel: Lustre: dolphin-MDT0000-mdc-ffff881065a37400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-1-33 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810264d1400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-0-18 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810343ce400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-1-46 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101d826800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-0-34 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881030f1ac00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:09:44 tscc-0-34 kernel: LustreError: Skipped 1 previous similar message
Aug 18 12:09:44 tscc-2-37 kernel: Lustre: dolphin-MDT0000-mdc-ffff881033c5a800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-0-21 kernel: Lustre: dolphin-MDT0000-mdc-ffff881032040400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-3-41 kernel: Lustre: dolphin-MDT0000-mdc-ffff882026020800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-1-54 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025a03800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-0-26 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810314d8800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-gpu-6-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476c19c00: Communicating with 172.25.32.25@tcp, operation mds_close failed with -107.
Aug 18 12:09:44 tscc-gpu-6-1 kernel: LustreError: Skipped 3 previous similar messages
Aug 18 12:09:44 tscc-gpu-6-1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880476c19c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 12:09:44 tscc-gpu-6-1 kernel: LustreError: 3636:0:(file.c:171:ll_close_inode_openhandle()) inode 144115538535121962 mdc close failed: rc = -5
Aug 18 12:09:44 tscc-gpu-6-1 kernel: LustreError: 10227:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 12:09:44 tscc-gpu-6-1 kernel: LustreError: 10227:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 28 previous similar messages
Aug 18 12:09:44 tscc-gpu-5-2 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff88086f941c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 12:09:44 tscc-gpu-5-7 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880876ea6c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 12:09:44 tscc-gpu-5-7 kernel: LustreError: 2168:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 12:09:44 tscc-gpu-5-7 kernel: LustreError: 2168:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 7 previous similar messages
Aug 18 12:09:44 tscc-gpu-5-2 kernel: LustreError: 28087:0:(file.c:171:ll_close_inode_openhandle()) inode 144115538568675350 mdc close failed: rc = -5
Aug 18 12:09:44 tscc-gpu-5-2 kernel: LustreError: 2179:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 12:09:44 tscc-gpu-5-2 kernel: LustreError: 28031:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 12:09:44 tscc-gpu-5-2 kernel: LustreError: 2179:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 2 previous similar messages
Aug 18 12:09:44 tscc-gpu-5-2 kernel: LustreError: 28031:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 2 previous similar messages
Aug 18 12:09:44 tscc-gpu-5-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876ea6c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-gpu-5-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff88086f941c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:44 tscc-gpu-6-1 kernel: LustreError: 10273:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005199:0x40b:0x0] error -108.
Aug 18 12:09:44 tscc-gpu-6-1 kernel: LustreError: 10275:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005199:0x3f3:0x0] error -108.
Aug 18 12:09:44 tscc-gpu-6-1 kernel: LustreError: 10275:0:(vvp_io.c:1229:vvp_io_init()) Skipped 28 previous similar messages
Aug 18 12:09:44 tscc-gpu-6-1 kernel: LustreError: 10273:0:(vvp_io.c:1229:vvp_io_init()) Skipped 37 previous similar messages
Aug 18 12:09:44 tscc-gpu-6-1 kernel: LustreError: 10707:0:(ldlm_resource.c:809:ldlm_resource_complain()) dolphin-MDT0000-mdc-ffff880476c19c00: namespace resource [0x200005199:0x40b:0x0].0 (ffff880875094840) refcount nonzero (0) after lock cleanup; forcing cleanup.
Aug 18 12:09:44 tscc-gpu-6-1 kernel: LustreError: 10707:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x200005199:0x40b:0x0].0 (ffff880875094840) refcount = 2
Aug 18 12:09:44 tscc-gpu-6-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476c19c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:09:50 tscc-1-43 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:09:52 tscc-pdafm-7-2 kernel: Lustre: 3151:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388985/real 1408388985]  req@ffff8810052e4c00 x1476802198378812/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388992 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:53 tscc-1-12 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:09:53 tscc-gpu-5-2 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:09:55 tscc-1-51 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388989/real 1408388989]  req@ffff881defc8f400 x1476542205956460/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388995 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:55 tscc-1-51 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:09:55 tscc-gpu-5-7 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:09:55 tscc-3-46 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:09:55 tscc-1-4 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:09:55 tscc-0-8 kernel: Lustre: 2748:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408388989/real 1408388989]  req@ffff880813195000 x1476547167692924/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408388995 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:09:56 tscc-1-38 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:09:57 tscc-0-29 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:09:58 tscc-2-11 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:09:58 tscc-2-11 kernel: Lustre: Skipped 1 previous similar message
Aug 18 12:10:04 tscc-gpu-3-2 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:04 tscc-3-41 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-gpu-6-6 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-gpu-3-4 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-0-63 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-0-60 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-2-55 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-pdafm-7-0 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-1-9 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-0-34 kernel: Lustre: dolphin-MDT0000-mdc-ffff881030f1ac00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-3-54 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-2-50 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-0-5 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-0-18 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-1-46 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-0-0 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-0-39 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-1-41 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-1-59 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-0-13 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-3-49 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-1-33 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-2-37 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:05 tscc-0-21 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:06 tscc-1-54 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:07 tscc-gpu-6-1 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:08 tscc-0-26 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:15 tscc-pdafm-7-2 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:17 tscc-0-8 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:10:17 tscc-1-51 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:11:16 tscc-3-56 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389069/real 1408389069]  req@ffff8810329fd000 x1476519829412956/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:11:16 tscc-3-56 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:11:16 tscc-3-56 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:11:16 tscc-0-31 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389069/real 1408389069]  req@ffff880816e75c00 x1476612858014800/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:11:16 tscc-0-31 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:11:16 tscc-1-32 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389069/real 1408389069]  req@ffff880ea9428000 x1476439898313304/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:11:16 tscc-1-32 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:11:16 tscc-1-32 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:11:16 tscc-0-28 kernel: Lustre: 2721:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389069/real 1408389069]  req@ffff880824095000 x1476547206532764/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:11:16 tscc-0-28 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:11:16 tscc-1-32 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:11:16 tscc-3-56 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:11:16 tscc-0-28 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:11:16 tscc-0-31 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:11:16 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 12:11:16 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:11:16 tscc-login1 kernel: LustreError: Skipped 1 previous similar message
Aug 18 12:11:16 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 12:11:16 tscc-login1 kernel: LustreError: 56313:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 12:11:16 tscc-login1 kernel: LustreError: 56313:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 68 previous similar messages
Aug 18 12:11:16 tscc-login1 kernel: LustreError: 5194:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff8810311a7800: Can't allocate new meta-sequence,rc -108
Aug 18 12:11:16 tscc-login1 kernel: LustreError: 5194:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff8810311a7800: Can't allocate new sequence, rc -108
Aug 18 12:11:16 tscc-login1 kernel: LustreError: 5194:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 12:11:16 tscc-login1 kernel: LustreError: 5376:0:(file.c:171:ll_close_inode_openhandle()) inode 144115527327990308 mdc close failed: rc = -108
Aug 18 12:11:16 tscc-login1 kernel: LustreError: 5376:0:(file.c:171:ll_close_inode_openhandle()) Skipped 2 previous similar messages
Aug 18 12:11:16 tscc-login1 kernel: LustreError: 56313:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID [0x250029c7:0x87004be5:0x0] error: rc = -108
Aug 18 12:11:16 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:11:17 tscc-0-7 kernel: Lustre: 2722:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389070/real 1408389070]  req@ffff88081b834800 x1476547271442044/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:11:17 tscc-0-7 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:11:17 tscc-0-7 kernel: LustreError: 11-0: MGC172.25.32.53@tcp: Communicating with 172.25.32.25@tcp, operation mgs_connect failed with -16.
Aug 18 12:11:42 tscc-0-7 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:16:13 tscc-gpu-6-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476c19c00: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 12:16:13 tscc-gpu-6-1 kernel: LustreError: Skipped 1 previous similar message
Aug 18 12:16:13 tscc-gpu-6-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476c19c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:16:13 tscc-gpu-5-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff88086f941c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:16:13 tscc-gpu-5-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876ea6c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:16:13 tscc-gpu-5-2 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff88086f941c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 12:16:13 tscc-gpu-5-7 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880876ea6c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 12:16:13 tscc-gpu-6-1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880476c19c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 12:16:13 tscc-gpu-6-1 kernel: LustreError: 3636:0:(file.c:171:ll_close_inode_openhandle()) inode 144115538619006978 mdc close failed: rc = -5
Aug 18 12:16:13 tscc-gpu-6-1 kernel: LustreError: 10227:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 12:16:13 tscc-gpu-6-1 kernel: LustreError: 10227:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 203 previous similar messages
Aug 18 12:16:13 tscc-gpu-5-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876ea6c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:16:13 tscc-gpu-5-2 kernel: Lustre: dolphin-MDT0000-mdc-ffff88086f941c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:16:13 tscc-gpu-6-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476c19c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:17:53 tscc-gpu-6-0 kernel: Lustre: Unmounted dolphin-client
Aug 18 12:19:53 tscc-gpu-6-0 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 12:19:56 tscc-gpu-6-5 kernel: Lustre: 21180:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389507/real 1408389507]  req@ffff8807dd8f9800 x1476570679056400/t0(0) o36->dolphin-MDT0000-mdc-ffff880479566400@172.25.32.25@tcp:12/10 lens 512/4328 e 2 to 1 dl 1408389596 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 12:19:56 tscc-gpu-6-5 kernel: Lustre: dolphin-MDT0000-mdc-ffff880479566400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:19:56 tscc-gpu-6-5 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880479566400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:19:56 tscc-gpu-6-5 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 12:19:58 tscc-gpu-6-0 kernel: Lustre: 2525:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389593/real 1408389593]  req@ffff88086a19fc00 x1476803525869572/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408389598 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:19:59 tscc-gpu-6-0 kernel: LustreError: 2461:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88086a19f800 x1476803525869576/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 12:20:19 tscc-gpu-6-5 kernel: Lustre: 13306:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389520/real 1408389520]  req@ffff880460efac00 x1476570679056664/t0(0) o101->dolphin-MDT0000-mdc-ffff880479566400@172.25.32.25@tcp:12/10 lens 376/1928 e 1 to 1 dl 1408389619 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 12:20:21 tscc-gpu-6-5 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880479566400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:20:21 tscc-1-7 kernel: Lustre: 10905:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389542/real 1408389542]  req@ffff880250c4c800 x1476519667324640/t0(0) o35->dolphin-MDT0000-mdc-ffff88101d811000@172.25.32.25@tcp:23/10 lens 392/4328 e 2 to 1 dl 1408389621 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 12:20:21 tscc-1-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101d811000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:20:21 tscc-1-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88101d811000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:20:23 tscc-gpu-6-0 kernel: Lustre: 2525:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389618/real 1408389618]  req@ffff8804721d8c00 x1476803525869664/t0(0) o38->dolphin-MDT0000-mdc-ffff88087989c400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408389623 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:20:43 tscc-gpu-6-0 kernel: Lustre: Layout lock feature supported.
Aug 18 12:20:43 tscc-gpu-6-0 kernel: Lustre: Mounted dolphin-client
Aug 18 12:20:46 tscc-gpu-6-5 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880479566400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:20:46 tscc-1-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88101d811000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:20:59 tscc-gpu-5-0 kernel: Lustre: 2580:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389560/real 1408389560]  req@ffff88046e20d400 x1476570047751392/t0(0) o101->dolphin-MDT0000-mdc-ffff880876e6d000@172.25.32.25@tcp:12/10 lens 592/2536 e 2 to 1 dl 1408389659 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 12:20:59 tscc-gpu-5-0 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876e6d000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:20:59 tscc-gpu-5-0 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876e6d000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:21:02 tscc-gpu-5-0 kernel: Lustre: 835:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389563/real 1408389563]  req@ffff88045e45fc00 x1476570047751672/t0(0) o36->dolphin-MDT0000-mdc-ffff880876e6d000@172.25.32.25@tcp:12/10 lens 512/4328 e 2 to 1 dl 1408389662 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 12:21:08 tscc-1-12 kernel: Lustre: 61119:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389524/real 1408389524]  req@ffff883e95672000 x1476547036135908/t0(0) o101->dolphin-MDT0000-mdc-ffff882011047000@172.25.32.25@tcp:12/10 lens 608/1136 e 2 to 1 dl 1408389668 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1
Aug 18 12:21:08 tscc-1-12 kernel: Lustre: dolphin-MDT0000-mdc-ffff882011047000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:21:08 tscc-1-12 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882011047000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:21:11 tscc-1-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88101d811000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:21:22 tscc-gpu-5-0 kernel: Lustre: 26467:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389576/real 1408389576]  req@ffff880433114000 x1476570047751980/t0(0) o101->dolphin-MDT0000-mdc-ffff880876e6d000@172.25.32.25@tcp:12/10 lens 376/1928 e 1 to 1 dl 1408389682 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 12:21:32 tscc-login1 kernel: Lustre: 2827:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389615/real 1408389615]  req@ffff880fde268c00 x1476710235549880/t0(0) o101->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.25@tcp:12/10 lens 592/2536 e 2 to 1 dl 1408389692 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 12:21:32 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 12:21:32 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:21:33 tscc-1-12 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882011047000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:21:36 tscc-gpu-6-5 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880479566400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:21:36 tscc-gpu-6-5 kernel: LustreError: Skipped 1 previous similar message
Aug 18 12:21:36 tscc-1-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88101d811000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:21:37 tscc-gpu-5-0 kernel: Lustre: 2694:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389591/real 1408389591]  req@ffff88085be35400 x1476570047752080/t0(0) o101->dolphin-MDT0000-mdc-ffff880876e6d000@172.25.32.25@tcp:12/10 lens 592/2536 e 1 to 1 dl 1408389697 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 12:21:49 tscc-gpu-5-0 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876e6d000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:21:49 tscc-gpu-5-0 kernel: LustreError: Skipped 1 previous similar message
Aug 18 12:21:57 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:21:58 tscc-1-12 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff882011047000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:22:01 tscc-1-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88101d811000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:22:23 tscc-1-12 kernel: Lustre: dolphin-MDT0000-mdc-ffff882011047000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:22:26 tscc-gpu-6-5 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880479566400: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 12:22:26 tscc-gpu-6-5 kernel: LustreError: 13306:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 12:22:26 tscc-gpu-6-5 kernel: LustreError: 13306:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 749 previous similar messages
Aug 18 12:22:26 tscc-gpu-6-5 kernel: LustreError: 13306:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005136:0xc80:0x0] error -5.
Aug 18 12:22:26 tscc-gpu-6-5 kernel: LustreError: 13306:0:(vvp_io.c:1229:vvp_io_init()) Skipped 717 previous similar messages
Aug 18 12:22:26 tscc-gpu-6-5 kernel: LustreError: 21688:0:(file.c:171:ll_close_inode_openhandle()) inode 144115536874177386 mdc close failed: rc = -108
Aug 18 12:22:26 tscc-gpu-6-5 kernel: Lustre: dolphin-MDT0000-mdc-ffff880479566400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:22:26 tscc-1-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff88101d811000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:22:39 tscc-gpu-5-0 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876e6d000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:22:47 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:23:09 tscc-login1 kernel: Lustre: 56313:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389647/real 1408389647]  req@ffff8804b8233800 x1476710235550424/t0(0) o36->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.25@tcp:12/10 lens 544/4328 e 1 to 1 dl 1408389789 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 12:23:09 tscc-login1 kernel: Lustre: 56313:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:23:12 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:23:37 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 12:24:02 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 12:24:02 tscc-login1 kernel: LustreError: 2827:0:(mdc_locks.c:1257:mdc_intent_getattr_async_interpret()) ldlm_cli_enqueue_fini: -5
Aug 18 12:24:02 tscc-login1 kernel: LustreError: 6280:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -108
Aug 18 12:24:02 tscc-login1 kernel: LustreError: 6280:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 15 previous similar messages
Aug 18 12:24:02 tscc-login1 kernel: LustreError: 2827:0:(mdc_locks.c:1257:mdc_intent_getattr_async_interpret()) Skipped 12 previous similar messages
Aug 18 12:24:02 tscc-login1 kernel: LustreError: 56313:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID [0x200005186:0xd:0x0] error: rc = -108
Aug 18 12:24:02 tscc-login1 kernel: LustreError: 6412:0:(file.c:171:ll_close_inode_openhandle()) inode 144115463390069864 mdc close failed: rc = -108
Aug 18 12:24:02 tscc-login1 kernel: LustreError: 6412:0:(file.c:171:ll_close_inode_openhandle()) Skipped 54 previous similar messages
Aug 18 12:24:02 tscc-login1 kernel: LustreError: 6280:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005186:0x6c:0x0] error -108.
Aug 18 12:24:02 tscc-login1 kernel: LustreError: 6280:0:(vvp_io.c:1229:vvp_io_init()) Skipped 60 previous similar messages
Aug 18 12:24:02 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 12:24:50 tscc-1-46 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389880/real 1408389880]  req@ffff880f71edb400 x1476536973631292/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:24:50 tscc-1-46 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:24:50 tscc-1-46 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:24:50 tscc-0-0 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389880/real 1408389880]  req@ffff880813e2d400 x1476519380636624/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:24:50 tscc-0-0 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:24:50 tscc-0-0 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:24:50 tscc-0-39 kernel: Lustre: 2228:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389880/real 1408389880]  req@ffff8803f464e000 x1475608471448248/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:24:50 tscc-0-39 kernel: Lustre: 2228:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:24:50 tscc-0-39 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:24:50 tscc-1-41 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389880/real 1408389880]  req@ffff880d33e3ac00 x1476537045952936/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:24:50 tscc-1-41 kernel: Lustre: 2731:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:24:50 tscc-1-41 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:24:50 tscc-1-59 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389880/real 1408389880]  req@ffff880691ada000 x1476536863877124/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:24:50 tscc-1-59 kernel: Lustre: 2733:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:24:50 tscc-1-59 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:24:50 tscc-0-13 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389880/real 1408389880]  req@ffff880817e75400 x1476634825426896/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:24:50 tscc-0-13 kernel: Lustre: 2720:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:24:50 tscc-0-13 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:24:50 tscc-1-54 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389881/real 1408389881]  req@ffff881017440400 x1476536988293708/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:24:50 tscc-1-54 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:24:50 tscc-1-54 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:24:50 tscc-3-49 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389880/real 1408389880]  req@ffff880fb1c8c000 x1476519923772448/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:24:50 tscc-3-49 kernel: Lustre: 2719:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:24:50 tscc-3-49 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:24:50 tscc-1-33 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389880/real 1408389880]  req@ffff88102582a400 x1476536972560328/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:24:50 tscc-1-33 kernel: Lustre: 2728:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:24:50 tscc-1-33 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:24:50 tscc-2-37 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389880/real 1408389880]  req@ffff880fb154f000 x1476570224103824/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:24:50 tscc-2-37 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:24:50 tscc-2-37 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:24:50 tscc-0-21 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408389880/real 1408389880]  req@ffff880820b5e000 x1476699991568148/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408389890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:24:50 tscc-0-21 kernel: Lustre: 2730:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 12:24:50 tscc-0-21 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 12:24:53 tscc-0-13 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:24:53 tscc-1-41 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:24:53 tscc-0-0 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:24:53 tscc-1-54 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:24:53 tscc-2-37 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:24:53 tscc-0-21 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:24:53 tscc-3-49 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:24:53 tscc-0-39 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:24:53 tscc-1-33 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:24:53 tscc-1-59 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:24:53 tscc-1-46 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 12:32:00 tscc-0-0 kernel: Lustre: Unmounted dolphin-client
Aug 18 12:32:30 tscc-0-0 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408390345/real 1408390345]  req@ffff881031dd2800 x1476519380643048/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408390350 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:32:31 tscc-0-0 kernel: LustreError: 55501:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff881031dd2c00 x1476519380643052/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 12:32:40 tscc-0-0 kernel: LustreError: 55514:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88081527f800 x1476519380643060/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 12:32:50 tscc-0-0 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1408390370/real 1408390370]  req@ffff880812306c00 x1476519380643144/t0(0) o38->dolphin-MDT0000-mdc-ffff881032959400@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408390375 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:33:15 tscc-0-0 kernel: Lustre: Layout lock feature supported.
Aug 18 12:33:16 tscc-0-0 kernel: Lustre: Mounted dolphin-client
Aug 18 12:34:33 tscc-0-62 kernel: Lustre: Unmounted dolphin-client
Aug 18 12:36:07 tscc-0-58 kernel: Lustre: Lustre: Build Version: v2_5_2_0-gf880594-CHANGED-2.6.32-431.23.3.el6.x86_64
Aug 18 12:36:13 tscc-0-58 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408390568/real 1408390568]  req@ffff88081fafbc00 x1476804548231172/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408390573 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:36:14 tscc-0-58 kernel: LustreError: 2651:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88081fafb800 x1476804548231176/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 12:36:20 tscc-0-58 kernel: LustreError: 2752:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88081fafb000 x1476804548231184/t0(0) o101->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Aug 18 12:36:38 tscc-0-58 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408390593/real 1408390593]  req@ffff88081fafb800 x1476804548231268/t0(0) o38->dolphin-MDT0000-mdc-ffff88083032fc00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408390598 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 12:36:58 tscc-0-58 kernel: Lustre: Layout lock feature supported.
Aug 18 12:36:58 tscc-0-58 kernel: Lustre: Mounted dolphin-client
Aug 18 13:03:55 tscc-login1 kernel: Lustre: 13173:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408392228/real 1408392228]  req@ffff880505a31c00 x1476710235640584/t0(0) o35->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408392235 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 13:03:55 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 13:04:02 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation ldlm_enqueue failed with -107.
Aug 18 13:04:02 tscc-0-15 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808300d3800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 13:04:02 tscc-0-15 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8808300d3800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 13:04:02 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 13:04:02 tscc-login1 kernel: LustreError: 2821:0:(mdc_locks.c:1257:mdc_intent_getattr_async_interpret()) ldlm_cli_enqueue_fini: -5
Aug 18 13:04:02 tscc-login1 kernel: LustreError: 13173:0:(file.c:171:ll_close_inode_openhandle()) inode 144115524375154985 mdc close failed: rc = -5
Aug 18 13:04:02 tscc-login1 kernel: LustreError: 13173:0:(file.c:171:ll_close_inode_openhandle()) Skipped 5 previous similar messages
Aug 18 13:04:02 tscc-login1 kernel: LustreError: 13173:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -108
Aug 18 13:04:02 tscc-login1 kernel: LustreError: 13173:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 65 previous similar messages
Aug 18 13:04:02 tscc-login1 kernel: LustreError: 2821:0:(mdc_locks.c:1257:mdc_intent_getattr_async_interpret()) Skipped 32 previous similar messages
Aug 18 13:04:02 tscc-0-15 kernel: LustreError: 25795:0:(file.c:171:ll_close_inode_openhandle()) inode 402159236 mdc close failed: rc = -108
Aug 18 13:04:02 tscc-0-15 kernel: LustreError: 25795:0:(file.c:171:ll_close_inode_openhandle()) Skipped 2 previous similar messages
Aug 18 13:04:02 tscc-0-15 kernel: Lustre: dolphin-MDT0000-mdc-ffff8808300d3800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 13:04:02 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 13:10:50 tscc-1-54 kernel: Lustre: 34817:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408392641/real 1408392641]  req@ffff881c74ff0000 x1476536988424624/t0(0) o35->dolphin-MDT0000-mdc-ffff881025a03800@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408392650 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 13:10:50 tscc-1-54 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025a03800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 13:10:50 tscc-gpu-5-2 kernel: Lustre: 2633:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408392643/real 1408392643]  req@ffff880849459400 x1476558499674228/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408392650 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 13:10:50 tscc-gpu-5-2 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 13:10:50 tscc-1-4 kernel: Lustre: 2740:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408392643/real 1408392643]  req@ffff880b3e142400 x1475918803030724/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408392650 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 13:10:50 tscc-1-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 13:10:50 tscc-gpu-5-7 kernel: Lustre: 2554:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408392643/real 1408392643]  req@ffff880869eed800 x1476612406367772/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408392650 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 13:10:50 tscc-gpu-5-7 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 13:10:50 tscc-3-46 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408392643/real 1408392643]  req@ffff880d76030800 x1476519915046708/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408392650 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 13:10:50 tscc-3-46 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 13:10:53 tscc-gpu-6-1 kernel: Lustre: 3636:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408392643/real 1408392643]  req@ffff8803e5976000 x1476558707141032/t0(0) o35->dolphin-MDT0000-mdc-ffff880476c19c00@172.25.32.25@tcp:23/10 lens 392/4328 e 0 to 1 dl 1408392653 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 13:10:53 tscc-gpu-6-1 kernel: Lustre: 3636:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Aug 18 13:10:53 tscc-gpu-6-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476c19c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 13:10:54 tscc-gpu-6-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476c19c00: Communicating with 172.25.32.25@tcp, operation mds_sync failed with -107.
Aug 18 13:10:54 tscc-gpu-6-1 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 13:10:54 tscc-gpu-5-7 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 13:10:54 tscc-gpu-5-2 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 13:10:54 tscc-3-46 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 13:10:54 tscc-1-54 kernel: Lustre: dolphin-MDT0000-mdc-ffff881025a03800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 13:10:54 tscc-gpu-6-1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880476c19c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 13:10:54 tscc-gpu-6-1 kernel: LustreError: 3636:0:(file.c:171:ll_close_inode_openhandle()) inode 144115538652561461 mdc close failed: rc = -5
Aug 18 13:10:54 tscc-gpu-6-1 kernel: LustreError: 10227:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 13:10:54 tscc-gpu-6-1 kernel: LustreError: 10227:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 1 previous similar message
Aug 18 13:10:54 tscc-gpu-6-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476c19c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 13:10:54 tscc-1-4 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 13:59:23 tscc-login1 kernel: Lustre: 33659:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395456/real 1408395456]  req@ffff8803b0bf4000 x1476710236342932/t0(0) o37->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.25@tcp:23/10 lens 448/440 e 2 to 1 dl 1408395563 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 13:59:23 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 13:59:23 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 13:59:23 tscc-login1 kernel: LustreError: Skipped 6 previous similar messages
Aug 18 13:59:28 tscc-login1 kernel: LustreError: 33659:0:(dir.c:422:ll_get_dir_page()) read cache page: [0x2000051a6:0x205:0x0] at 0: rc -4
Aug 18 13:59:28 tscc-login1 kernel: LustreError: 33659:0:(dir.c:584:ll_dir_read()) error reading dir [0x2000051a6:0x205:0x0] at 0: rc -4
Aug 18 13:59:31 tscc-login1 kernel: LustreError: 34740:0:(dir.c:398:ll_get_dir_page()) dir page locate: [0x2000051a6:0x205:0x0] at 0: rc -5
Aug 18 13:59:31 tscc-login1 kernel: LustreError: 34740:0:(dir.c:584:ll_dir_read()) error reading dir [0x2000051a6:0x205:0x0] at 0: rc -5
Aug 18 13:59:32 tscc-login1 kernel: LustreError: 34741:0:(dir.c:398:ll_get_dir_page()) dir page locate: [0x2000051a6:0x205:0x0] at 0: rc -5
Aug 18 13:59:32 tscc-login1 kernel: LustreError: 34741:0:(dir.c:584:ll_dir_read()) error reading dir [0x2000051a6:0x205:0x0] at 0: rc -5
Aug 18 13:59:48 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:00:05 tscc-login1 kernel: LustreError: 34823:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -4
Aug 18 14:00:05 tscc-login1 kernel: LustreError: 34823:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 71 previous similar messages
Aug 18 14:00:13 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:00:38 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:00:40 tscc-gpu-6-7 kernel: Lustre: 2536:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395563/real 1408395563]  req@ffff880709897800 x1476609758156304/t0(0) o101->dolphin-MDT0000-mdc-ffff88087948ec00@172.25.32.25@tcp:12/10 lens 592/2536 e 2 to 1 dl 1408395640 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 14:00:40 tscc-gpu-6-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff88087948ec00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 14:00:40 tscc-gpu-6-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88087948ec00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:00:40 tscc-gpu-6-7 kernel: LustreError: Skipped 4 previous similar messages
Aug 18 14:00:47 tscc-gpu-6-1 kernel: Lustre: 10224:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395570/real 1408395570]  req@ffff88040a802c00 x1476558707273892/t0(0) o36->dolphin-MDT0000-mdc-ffff880476c19c00@172.25.32.25@tcp:12/10 lens 544/4328 e 2 to 1 dl 1408395647 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 14:00:47 tscc-gpu-6-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476c19c00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 14:00:47 tscc-gpu-6-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476c19c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:00:47 tscc-gpu-6-1 kernel: LustreError: Skipped 3 previous similar messages
Aug 18 14:00:54 tscc-gpu-5-3 kernel: Lustre: 1274:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395563/real 1408395563]  req@ffff8804244cc400 x1476570130415468/t0(0) o36->dolphin-MDT0000-mdc-ffff880876924800@172.25.32.25@tcp:12/10 lens 544/4328 e 1 to 1 dl 1408395654 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 14:00:54 tscc-gpu-5-3 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876924800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 14:00:54 tscc-gpu-5-3 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876924800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:01:00 tscc-gpu-5-0 kernel: Lustre: 2575:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395568/real 1408395568]  req@ffff88045e78a800 x1476570048042256/t0(0) o101->dolphin-MDT0000-mdc-ffff880876e6d000@172.25.32.25@tcp:12/10 lens 592/2536 e 1 to 1 dl 1408395659 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 14:01:00 tscc-gpu-5-0 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876e6d000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 14:01:00 tscc-gpu-5-0 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876e6d000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:01:00 tscc-gpu-5-0 kernel: LustreError: Skipped 1 previous similar message
Aug 18 14:01:03 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:01:12 tscc-gpu-6-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476c19c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:01:19 tscc-gpu-5-3 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876924800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:01:25 tscc-gpu-5-0 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876e6d000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:01:28 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:01:30 tscc-gpu-6-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88087948ec00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:01:30 tscc-gpu-6-7 kernel: LustreError: Skipped 1 previous similar message
Aug 18 14:01:37 tscc-gpu-6-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476c19c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:01:44 tscc-gpu-5-3 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876924800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:01:50 tscc-gpu-5-0 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876e6d000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:01:53 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 14:01:53 tscc-login1 kernel: LustreError: 55183:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 14:01:53 tscc-login1 kernel: LustreError: 40467:0:(dir.c:378:ll_get_dir_page()) lock enqueue: [0x14e043f2:0xa0265a04:0x0] at 0: rc -5
Aug 18 14:01:53 tscc-login1 kernel: LustreError: 40467:0:(dir.c:584:ll_dir_read()) error reading dir [0x14e043f2:0xa0265a04:0x0] at 0: rc -5
Aug 18 14:01:53 tscc-login1 kernel: LustreError: 34723:0:(dir.c:422:ll_get_dir_page()) read cache page: [0x2000051a6:0xf2:0x0] at 0: rc -5
Aug 18 14:01:53 tscc-login1 kernel: LustreError: 55183:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 8 previous similar messages
Aug 18 14:01:53 tscc-login1 kernel: LustreError: 55183:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x2000051a6:0x233:0x0] error -5.
Aug 18 14:01:53 tscc-login1 kernel: LustreError: 55183:0:(vvp_io.c:1229:vvp_io_init()) Skipped 45 previous similar messages
Aug 18 14:01:53 tscc-login1 kernel: LustreError: 35243:0:(file.c:171:ll_close_inode_openhandle()) inode 144115463390069478 mdc close failed: rc = -108
Aug 18 14:01:53 tscc-login1 kernel: LustreError: 35243:0:(file.c:171:ll_close_inode_openhandle()) Skipped 15 previous similar messages
Aug 18 14:01:53 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 14:02:08 tscc-gpu-6-1 kernel: Lustre: 2523:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395722/real 1408395722]  req@ffff88086d0fe400 x1476558707275880/t0(0) o38->dolphin-MDT0000-mdc-ffff880476c19c00@172.25.32.25@tcp:12/10 lens 400/544 e 0 to 1 dl 1408395728 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:02:09 tscc-gpu-5-3 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876924800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:02:12 tscc-1-54 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395725/real 1408395725]  req@ffff881fed05bc00 x1476536988637868/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408395732 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:02:12 tscc-1-54 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 14:02:12 tscc-3-46 kernel: Lustre: 2732:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395725/real 1408395725]  req@ffff880ee5c7d800 x1476519915102600/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408395732 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:02:12 tscc-3-46 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 14:02:12 tscc-1-4 kernel: Lustre: 2746:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395725/real 1408395725]  req@ffff880bd72ea800 x1475918803202892/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408395732 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:02:12 tscc-1-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 14:02:13 tscc-gpu-5-0 kernel: Lustre: 830:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395591/real 1408395591]  req@ffff88082cad4c00 x1476570048043156/t0(0) o36->dolphin-MDT0000-mdc-ffff880876e6d000@172.25.32.25@tcp:12/10 lens 544/4328 e 1 to 1 dl 1408395733 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 14:02:15 tscc-gpu-6-7 kernel: Lustre: 12399:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395593/real 1408395593]  req@ffff88085b2d0800 x1476609758157572/t0(0) o101->dolphin-MDT0000-mdc-ffff88087948ec00@172.25.32.25@tcp:12/10 lens 592/2536 e 1 to 1 dl 1408395735 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 14:02:17 tscc-gpu-6-6 kernel: Lustre: 2528:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395730/real 1408395730]  req@ffff88044fdbe800 x1476558758479320/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408395737 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:02:17 tscc-gpu-6-6 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 14:02:17 tscc-0-63 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395730/real 1408395730]  req@ffff880818657c00 x1476798012576260/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408395737 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:02:17 tscc-0-63 kernel: Lustre: 2726:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 14:02:17 tscc-0-63 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 14:02:17 tscc-0-60 kernel: Lustre: 2823:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395730/real 1408395730]  req@ffff881033c93000 x1476700762774948/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408395737 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:02:17 tscc-0-60 kernel: Lustre: 2823:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 14:02:17 tscc-0-60 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 14:02:17 tscc-0-18 kernel: Lustre: 2769:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395730/real 1408395730]  req@ffff880832b47800 x1476547249572988/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408395737 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:02:17 tscc-0-18 kernel: Lustre: 2769:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 14:02:17 tscc-0-18 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 14:02:18 tscc-1-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395732/real 1408395732]  req@ffff881fe4deb400 x1476536988638072/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408395738 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:02:18 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395732/real 1408395732]  req@ffff880de2e4a400 x1476519915102872/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 400/544 e 0 to 1 dl 1408395738 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:02:18 tscc-gpu-6-6 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 14:02:18 tscc-0-60 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 14:02:18 tscc-0-18 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 14:02:18 tscc-0-63 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 14:02:18 tscc-1-4 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 14:02:33 tscc-gpu-6-1 kernel: Lustre: 2523:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395747/real 1408395747]  req@ffff880845a47c00 x1476558707276144/t0(0) o38->dolphin-MDT0000-mdc-ffff880476c19c00@172.25.32.53@tcp:12/10 lens 400/544 e 0 to 1 dl 1408395753 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:02:34 tscc-gpu-5-3 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876924800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:02:40 tscc-gpu-5-0 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876e6d000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:02:40 tscc-gpu-5-0 kernel: LustreError: Skipped 1 previous similar message
Aug 18 14:02:43 tscc-1-54 kernel: Lustre: 2718:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395757/real 1408395757]  req@ffff881ff2015800 x1476536988639736/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408395763 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:02:43 tscc-3-46 kernel: Lustre: 2717:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395757/real 1408395757]  req@ffff880e92b3a000 x1476519915103272/t0(0) o250->MGC172.25.32.53@tcp@172.25.32.53@tcp:26/25 lens 400/544 e 0 to 1 dl 1408395763 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:02:47 tscc-gpu-5-0 kernel: Lustre: 6077:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395598/real 1408395598]  req@ffff880857881800 x1476570048043264/t0(0) o101->dolphin-MDT0000-mdc-ffff880876e6d000@172.25.32.25@tcp:12/10 lens 592/2536 e 1 to 1 dl 1408395767 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 14:02:52 tscc-gpu-6-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476c19c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:03:02 tscc-1-54 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 14:03:02 tscc-3-46 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 14:03:07 tscc-gpu-5-6 kernel: Lustre: 28192:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395618/real 1408395618]  req@ffff88044718ec00 x1476570264410032/t0(0) o36->dolphin-MDT0000-mdc-ffff880476fccc00@172.25.32.25@tcp:12/10 lens 544/4328 e 1 to 1 dl 1408395787 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 14:03:07 tscc-gpu-5-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476fccc00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 14:03:07 tscc-gpu-5-6 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476fccc00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:03:10 tscc-gpu-6-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88087948ec00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:03:10 tscc-gpu-6-7 kernel: LustreError: Skipped 3 previous similar messages
Aug 18 14:03:17 tscc-gpu-6-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476c19c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:03:24 tscc-gpu-5-3 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880876924800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 14:03:24 tscc-gpu-5-3 kernel: LustreError: 27449:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -108
Aug 18 14:03:24 tscc-gpu-5-3 kernel: LustreError: 27449:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 708 previous similar messages
Aug 18 14:03:24 tscc-gpu-5-3 kernel: LustreError: 1274:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff880876924800: Can't allocate new meta-sequence,rc -108
Aug 18 14:03:24 tscc-gpu-5-3 kernel: LustreError: 1274:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff880876924800: Can't allocate new sequence, rc -108
Aug 18 14:03:24 tscc-gpu-5-3 kernel: LustreError: 1274:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 14:03:24 tscc-gpu-5-3 kernel: LustreError: 1271:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005138:0x14c9:0x0] error -108.
Aug 18 14:03:24 tscc-gpu-5-3 kernel: LustreError: 1271:0:(vvp_io.c:1229:vvp_io_init()) Skipped 696 previous similar messages
Aug 18 14:03:24 tscc-gpu-5-3 kernel: LustreError: 6217:0:(file.c:171:ll_close_inode_openhandle()) inode 144115536907731467 mdc close failed: rc = -108
Aug 18 14:03:24 tscc-gpu-5-3 kernel: LustreError: 1274:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID [0x250029c7:0x87004be5:0x0] error: rc = -108
Aug 18 14:03:24 tscc-gpu-5-3 kernel: LustreError: 6217:0:(ldlm_resource.c:809:ldlm_resource_complain()) dolphin-MDT0000-mdc-ffff880876924800: namespace resource [0x200005138:0x1489:0x0].0 (ffff88044cfea980) refcount nonzero (0) after lock cleanup; forcing cleanup.
Aug 18 14:03:24 tscc-gpu-5-3 kernel: LustreError: 6217:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x200005138:0x1489:0x0].0 (ffff88044cfea980) refcount = 2
Aug 18 14:03:24 tscc-gpu-5-3 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876924800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 14:03:34 tscc-gpu-3-1 kernel: Lustre: 12846:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395704/real 1408395704]  req@ffff8804751e7400 x1476802125061032/t0(0) o36->dolphin-MDT0000-mdc-ffff880876650000@172.25.32.25@tcp:12/10 lens 528/4328 e 1 to 1 dl 1408395814 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 14:03:34 tscc-gpu-3-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876650000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 14:03:34 tscc-gpu-3-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876650000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:03:35 tscc-gpu-6-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff88087948ec00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 14:03:42 tscc-gpu-6-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476c19c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:03:55 tscc-gpu-5-0 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876e6d000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:03:55 tscc-gpu-5-0 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 14:03:59 tscc-gpu-3-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876650000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:04:07 tscc-gpu-6-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476c19c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:04:24 tscc-gpu-3-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876650000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:04:47 tscc-gpu-5-6 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476fccc00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:04:47 tscc-gpu-5-6 kernel: LustreError: Skipped 3 previous similar messages
Aug 18 14:04:49 tscc-gpu-3-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876650000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:04:52 tscc-gpu-6-7 kernel: Lustre: 2536:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395815/real 1408395815]  req@ffff880709897800 x1476609758156304/t0(0) o101->dolphin-MDT0000-mdc-ffff88087948ec00@172.25.32.25@tcp:12/10 lens 592/2536 e 2 to 1 dl 1408395892 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Aug 18 14:04:52 tscc-gpu-6-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff88087948ec00: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 14:04:57 tscc-gpu-6-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476c19c00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:04:57 tscc-gpu-6-1 kernel: LustreError: Skipped 1 previous similar message
Aug 18 14:05:14 tscc-gpu-3-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876650000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:05:22 tscc-gpu-6-1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880476c19c00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 14:05:22 tscc-gpu-6-1 kernel: LustreError: 3676:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -108
Aug 18 14:05:22 tscc-gpu-6-1 kernel: LustreError: 10274:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff880476c19c00: Can't allocate new meta-sequence,rc -108
Aug 18 14:05:22 tscc-gpu-6-1 kernel: LustreError: 10274:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff880476c19c00: Can't allocate new sequence, rc -108
Aug 18 14:05:22 tscc-gpu-6-1 kernel: LustreError: 10274:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 14:05:22 tscc-gpu-6-1 kernel: LustreError: 10259:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005199:0x3ab:0x0] error -108.
Aug 18 14:05:22 tscc-gpu-6-1 kernel: LustreError: 10259:0:(vvp_io.c:1229:vvp_io_init()) Skipped 148 previous similar messages
Aug 18 14:05:22 tscc-gpu-6-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476c19c00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 14:05:22 tscc-gpu-6-1 kernel: LustreError: 3676:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 7 previous similar messages
Aug 18 14:05:39 tscc-gpu-3-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876650000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:05:42 tscc-gpu-6-7 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff88087948ec00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:05:42 tscc-gpu-6-7 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 14:06:04 tscc-gpu-3-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876650000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:06:25 tscc-gpu-5-0 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876e6d000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:06:25 tscc-gpu-5-0 kernel: LustreError: Skipped 5 previous similar messages
Aug 18 14:06:54 tscc-gpu-3-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876650000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:06:54 tscc-gpu-3-1 kernel: LustreError: Skipped 1 previous similar message
Aug 18 14:06:58 tscc-0-31 kernel: Lustre: 62334:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395814/real 1408395814]  req@ffff88102c0bec00 x1476612858105356/t0(0) o36->dolphin-MDT0000-mdc-ffff880830a6b000@172.25.32.25@tcp:12/10 lens 536/4328 e 1 to 1 dl 1408396018 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 14:06:58 tscc-0-31 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830a6b000: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 14:06:58 tscc-0-31 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880830a6b000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:07:23 tscc-0-31 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880830a6b000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:07:26 tscc-gpu-6-5 kernel: Lustre: 22738:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395719/real 1408395719]  req@ffff88043dcb3400 x1476570679255528/t0(0) o36->dolphin-MDT0000-mdc-ffff880479566400@172.25.32.25@tcp:12/10 lens 544/4328 e 1 to 1 dl 1408396046 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 14:07:26 tscc-gpu-6-5 kernel: Lustre: dolphin-MDT0000-mdc-ffff880479566400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 14:07:26 tscc-gpu-6-5 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880479566400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:07:26 tscc-gpu-6-5 kernel: LustreError: Skipped 1 previous similar message
Aug 18 14:07:42 tscc-gpu-5-6 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880476fccc00: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:07:42 tscc-gpu-5-6 kernel: LustreError: Skipped 6 previous similar messages
Aug 18 14:07:48 tscc-0-31 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880830a6b000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:07:51 tscc-gpu-6-5 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880479566400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:08:00 tscc-0-29 kernel: Lustre: 2727:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408396072/real 1408396072]  req@ffff8803b2509400 x1476612872678240/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408396080 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:08:00 tscc-0-29 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 14:08:03 tscc-0-29 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 14:08:12 tscc-gpu-3-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876650000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:08:12 tscc-gpu-3-1 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 14:08:16 tscc-gpu-6-5 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880479566400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:08:18 tscc-0-31 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880830a6b000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:08:28 tscc-login2 kernel: Lustre: 40186:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395781/real 1408395781]  req@ffff880adbff1400 x1476716826843016/t0(0) o36->dolphin-MDT0000-mdc-ffff881033c5d400@172.25.32.25@tcp:12/10 lens 536/4328 e 1 to 1 dl 1408396108 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 14:08:28 tscc-login2 kernel: Lustre: dolphin-MDT0000-mdc-ffff881033c5d400: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 14:08:28 tscc-login2 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881033c5d400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:08:39 tscc-0-31 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880830a6b000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:08:53 tscc-login2 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881033c5d400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:08:58 tscc-gpu-5-4 kernel: Lustre: 2621:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408396131/real 1408396131]  req@ffff880463d6f000 x1476612338182012/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408396138 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:08:58 tscc-gpu-5-4 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 14:08:58 tscc-1-6 kernel: Lustre: 2734:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408396131/real 1408396131]  req@ffff8807c587f000 x1476519511884284/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408396138 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:08:58 tscc-1-6 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 14:09:02 tscc-gpu-6-7 kernel: Lustre: 12399:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395815/real 1408395815]  req@ffff88085b2d0800 x1476609758157572/t0(0) o101->dolphin-MDT0000-mdc-ffff88087948ec00@172.25.32.25@tcp:12/10 lens 592/2536 e 2 to 1 dl 1408396142 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Aug 18 14:09:02 tscc-gpu-5-4 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 14:09:02 tscc-1-6 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 14:09:03 tscc-0-31 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880830a6b000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:09:06 tscc-gpu-6-5 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880479566400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:09:06 tscc-gpu-6-5 kernel: LustreError: Skipped 1 previous similar message
Aug 18 14:09:18 tscc-login2 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881033c5d400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:09:27 tscc-gpu-6-7 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff88087948ec00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 14:09:27 tscc-gpu-6-7 kernel: LustreError: 12399:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 14:09:27 tscc-gpu-6-7 kernel: LustreError: 12399:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 7 previous similar messages
Aug 18 14:09:27 tscc-gpu-6-7 kernel: LustreError: 7245:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff88087948ec00: Can't allocate new meta-sequence,rc -108
Aug 18 14:09:27 tscc-gpu-6-7 kernel: LustreError: 7245:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff88087948ec00: Can't allocate new sequence, rc -108
Aug 18 14:09:27 tscc-gpu-6-7 kernel: LustreError: 7245:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 14:09:27 tscc-gpu-6-7 kernel: LustreError: 2531:0:(mdc_locks.c:1257:mdc_intent_getattr_async_interpret()) ldlm_cli_enqueue_fini: -108
Aug 18 14:09:27 tscc-gpu-6-7 kernel: LustreError: 7245:0:(mdc_locks.c:1209:mdc_intent_lock()) Skipped 1 previous similar message
Aug 18 14:09:27 tscc-gpu-6-7 kernel: LustreError: 777:0:(file.c:171:ll_close_inode_openhandle()) inode 144115536958065411 mdc close failed: rc = -108
Aug 18 14:09:27 tscc-gpu-6-7 kernel: LustreError: 2531:0:(mdc_locks.c:1257:mdc_intent_getattr_async_interpret()) Skipped 3 previous similar messages
Aug 18 14:09:27 tscc-gpu-6-7 kernel: LustreError: 12399:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x20000513b:0x158:0x0] error -108.
Aug 18 14:09:27 tscc-gpu-6-7 kernel: LustreError: 12399:0:(vvp_io.c:1229:vvp_io_init()) Skipped 14 previous similar messages
Aug 18 14:09:27 tscc-gpu-6-7 kernel: Lustre: dolphin-MDT0000-mdc-ffff88087948ec00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 14:09:33 tscc-0-31 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880830a6b000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:09:39 tscc-1-35 kernel: Lustre: 2725:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408396170/real 1408396170]  req@ffff880692d7ac00 x1476285713931852/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408396179 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:09:39 tscc-1-35 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 14:09:39 tscc-3-40 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408396170/real 1408396170]  req@ffff880e939b3000 x1476520324521688/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408396179 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:09:39 tscc-3-40 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Aug 18 14:09:39 tscc-3-40 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 14:09:39 tscc-1-56 kernel: Lustre: 2729:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408396170/real 1408396170]  req@ffff882028171400 x1476536956541572/t0(0) o400->MGC172.25.32.53@tcp@172.25.32.25@tcp:26/25 lens 224/224 e 0 to 1 dl 1408396179 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Aug 18 14:09:39 tscc-1-56 kernel: LustreError: 166-1: MGC172.25.32.53@tcp: Connection to MGS (at 172.25.32.25@tcp) was lost; in progress operations using this service will fail
Aug 18 14:09:42 tscc-1-56 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 14:09:42 tscc-1-35 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 14:09:42 tscc-3-40 kernel: Lustre: MGC172.25.32.53@tcp: Connection restored to MGS (at 172.25.32.25@tcp)
Aug 18 14:09:43 tscc-login2 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881033c5d400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880876e6d000: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 2575:0:(mdc_locks.c:1257:mdc_intent_getattr_async_interpret()) ldlm_cli_enqueue_fini: -5
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 6077:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 6077:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 507 previous similar messages
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 897:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff880876e6d000: Can't allocate new meta-sequence,rc -108
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 897:0:(fid_request.c:220:seq_client_alloc_seq()) Skipped 1 previous similar message
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 897:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff880876e6d000: Can't allocate new sequence, rc -108
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 897:0:(fid_request.c:354:seq_client_alloc_fid()) Skipped 1 previous similar message
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 897:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 897:0:(mdc_locks.c:1209:mdc_intent_lock()) Skipped 1 previous similar message
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 2575:0:(mdc_locks.c:1257:mdc_intent_getattr_async_interpret()) Skipped 3 previous similar messages
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 6408:0:(file.c:171:ll_close_inode_openhandle()) inode 144115536941285894 mdc close failed: rc = -108
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 6077:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x20000513a:0x270:0x0] error -108.
Aug 18 14:09:45 tscc-gpu-5-0 kernel: LustreError: 6077:0:(vvp_io.c:1229:vvp_io_init()) Skipped 470 previous similar messages
Aug 18 14:09:45 tscc-gpu-5-0 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876e6d000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 14:10:08 tscc-login2 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881033c5d400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:10:18 tscc-0-31 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880830a6b000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:10:18 tscc-0-31 kernel: LustreError: Skipped 1 previous similar message
Aug 18 14:10:21 tscc-gpu-6-5 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880479566400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:10:21 tscc-gpu-6-5 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 14:10:33 tscc-login2 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff881033c5d400: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:10:39 tscc-gpu-3-1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880876650000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:10:39 tscc-gpu-3-1 kernel: LustreError: Skipped 5 previous similar messages
Aug 18 14:10:41 tscc-login2 kernel: LustreError: 42213:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -4
Aug 18 14:10:47 tscc-login1 kernel: Lustre: 35738:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1408395848/real 1408395848]  req@ffff880a225fb400 x1476710236414156/t0(0) o36->dolphin-MDT0000-mdc-ffff8810311a7800@172.25.32.25@tcp:12/10 lens 528/4328 e 1 to 1 dl 1408396247 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Aug 18 14:10:47 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection to dolphin-MDT0000 (at 172.25.32.25@tcp) was lost; in progress operations using this service will wait for recovery to complete
Aug 18 14:10:47 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:11:11 tscc-gpu-6-5 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880479566400: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 14:11:11 tscc-gpu-6-5 kernel: LustreError: 13551:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -108
Aug 18 14:11:11 tscc-gpu-6-5 kernel: LustreError: 13551:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 48 previous similar messages
Aug 18 14:11:11 tscc-gpu-6-5 kernel: LustreError: 22729:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff880479566400: Can't allocate new meta-sequence,rc -108
Aug 18 14:11:11 tscc-gpu-6-5 kernel: LustreError: 22729:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff880479566400: Can't allocate new sequence, rc -108
Aug 18 14:11:11 tscc-gpu-6-5 kernel: LustreError: 22729:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 14:11:11 tscc-gpu-6-5 kernel: Lustre: dolphin-MDT0000-mdc-ffff880479566400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 14:11:16 tscc-login2 kernel: LustreError: 42554:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -4
Aug 18 14:11:23 tscc-login2 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff881033c5d400: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 14:11:23 tscc-login2 kernel: LustreError: 42582:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 14:11:23 tscc-login2 kernel: LustreError: 41442:0:(dir.c:378:ll_get_dir_page()) lock enqueue: [0x250029c7:0x87004be5:0x0] at 0: rc -108
Aug 18 14:11:23 tscc-login2 kernel: LustreError: 41464:0:(dir.c:422:ll_get_dir_page()) read cache page: [0x200004e8a:0x14ed:0x0] at 0: rc -108
Aug 18 14:11:23 tscc-login2 kernel: LustreError: 41464:0:(dir.c:584:ll_dir_read()) error reading dir [0x200004e8a:0x14ed:0x0] at 0: rc -108
Aug 18 14:11:23 tscc-login2 kernel: LustreError: 42583:0:(file.c:171:ll_close_inode_openhandle()) inode 144115536756740852 mdc close failed: rc = -108
Aug 18 14:11:23 tscc-login2 kernel: LustreError: 42583:0:(file.c:171:ll_close_inode_openhandle()) Skipped 5 previous similar messages
Aug 18 14:11:23 tscc-login2 kernel: LustreError: 40186:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff881033c5d400: Can't allocate new meta-sequence,rc -108
Aug 18 14:11:23 tscc-login2 kernel: LustreError: 40186:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff881033c5d400: Can't allocate new sequence, rc -108
Aug 18 14:11:23 tscc-login2 kernel: LustreError: 40186:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 14:11:23 tscc-login2 kernel: Lustre: dolphin-MDT0000-mdc-ffff881033c5d400: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 14:11:27 tscc-gpu-5-6 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880476fccc00: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 14:11:27 tscc-gpu-5-6 kernel: LustreError: 19575:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -108
Aug 18 14:11:27 tscc-gpu-5-6 kernel: LustreError: 19575:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 27 previous similar messages
Aug 18 14:11:27 tscc-gpu-5-6 kernel: LustreError: 28183:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff880476fccc00: Can't allocate new meta-sequence,rc -108
Aug 18 14:11:27 tscc-gpu-5-6 kernel: LustreError: 31206:0:(file.c:171:ll_close_inode_openhandle()) inode 144115536924509031 mdc close failed: rc = -108
Aug 18 14:11:27 tscc-gpu-5-6 kernel: LustreError: 28183:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff880476fccc00: Can't allocate new sequence, rc -108
Aug 18 14:11:27 tscc-gpu-5-6 kernel: LustreError: 28183:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 14:11:27 tscc-gpu-5-6 kernel: LustreError: 28183:0:(mdc_locks.c:1209:mdc_intent_lock()) Skipped 1 previous similar message
Aug 18 14:11:27 tscc-gpu-5-6 kernel: LustreError: 31207:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID [0x250029c7:0x87004be5:0x0] error: rc = -108
Aug 18 14:11:27 tscc-gpu-5-6 kernel: Lustre: dolphin-MDT0000-mdc-ffff880476fccc00: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 14:11:33 tscc-0-31 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880830a6b000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:11:33 tscc-0-31 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 14:11:37 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:11:37 tscc-login1 kernel: LustreError: Skipped 1 previous similar message
Aug 18 14:12:52 tscc-login1 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff8810311a7800: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:12:52 tscc-login1 kernel: LustreError: Skipped 2 previous similar messages
Aug 18 14:13:09 tscc-gpu-3-1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880876650000: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 14:13:09 tscc-gpu-3-1 kernel: LustreError: 12846:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -108
Aug 18 14:13:09 tscc-gpu-3-1 kernel: LustreError: 12846:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff880876650000: Can't allocate new meta-sequence,rc -108
Aug 18 14:13:09 tscc-gpu-3-1 kernel: Lustre: dolphin-MDT0000-mdc-ffff880876650000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 14:13:09 tscc-gpu-3-1 kernel: LustreError: 12846:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff880876650000: Can't allocate new sequence, rc -108
Aug 18 14:13:09 tscc-gpu-3-1 kernel: LustreError: 12846:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff8810311a7800: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 62466:0:(file.c:171:ll_close_inode_openhandle()) inode 144115538283463013 mdc close failed: rc = -5
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 42703:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -5
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 42703:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 1 previous similar message
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 56399:0:(fid_request.c:220:seq_client_alloc_seq()) cli-cli-dolphin-MDT0000-mdc-ffff8810311a7800: Can't allocate new meta-sequence,rc -108
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 56399:0:(fid_request.c:354:seq_client_alloc_fid()) cli-cli-dolphin-MDT0000-mdc-ffff8810311a7800: Can't allocate new sequence, rc -108
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 56399:0:(mdc_locks.c:1209:mdc_intent_lock()) Can't alloc new fid, rc -108
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 56399:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file layout [0x200005186:0xc:0x0] error -108.
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 56399:0:(vvp_io.c:1229:vvp_io_init()) Skipped 1 previous similar message
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 62466:0:(file.c:171:ll_close_inode_openhandle()) Skipped 35 previous similar messages
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 37387:0:(ldlm_resource.c:809:ldlm_resource_complain()) dolphin-MDT0000-mdc-ffff8810311a7800: namespace resource [0x200005186:0xc:0x0].0 (ffff88029cb30480) refcount nonzero (1) after lock cleanup; forcing cleanup.
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 37387:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x200005186:0xc:0x0].0 (ffff88029cb30480) refcount = 2
Aug 18 14:13:17 tscc-login1 kernel: Lustre: dolphin-MDT0000-mdc-ffff8810311a7800: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)
Aug 18 14:14:03 tscc-0-31 kernel: LustreError: 11-0: dolphin-MDT0000-mdc-ffff880830a6b000: Communicating with 172.25.32.25@tcp, operation mds_connect failed with -16.
Aug 18 14:14:03 tscc-0-31 kernel: LustreError: Skipped 5 previous similar messages
Aug 18 14:14:35 tscc-gpu-5-0 kernel: LustreError: 11-0: dolphin-OST0035-osc-ffff880876e6d000: Communicating with 172.25.33.245@tcp, operation ldlm_enqueue failed with -12.
Aug 18 14:14:35 tscc-gpu-5-0 kernel: LustreError: Skipped 7 previous similar messages
Aug 18 14:14:53 tscc-0-31 kernel: LustreError: 167-0: dolphin-MDT0000-mdc-ffff880830a6b000: This client was evicted by dolphin-MDT0000; in progress operations using this service will fail.
Aug 18 14:14:53 tscc-0-31 kernel: LustreError: 62790:0:(file.c:171:ll_close_inode_openhandle()) inode 402159236 mdc close failed: rc = -108
Aug 18 14:14:53 tscc-0-31 kernel: LustreError: 61779:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -108
Aug 18 14:14:53 tscc-0-31 kernel: Lustre: dolphin-MDT0000-mdc-ffff880830a6b000: Connection restored to dolphin-MDT0000 (at 172.25.32.25@tcp)