Jun 6 03:04:47 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client 95baf44a-85d2-9f20-96ae-efb7b93cffe2 (at 11787@ptl1) in 1352 seconds. I think it's dead, and I am evicting it. Jun 6 07:30:01 widow-mds3 kernel: Lustre: Client widow3-client has started Jun 6 09:54:54 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client 64534329-044c-ebaa-cdaa-96841bf24710 (at 3591@ptl1) in 1352 seconds. I think it's dead, and I am evicting it. Jun 6 09:54:54 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client adbccfa5-3e2d-2719-3e43-92b67318a14a (at 10869@ptl1) in 1352 seconds. I think it's dead, and I am evicting it. Jun 6 09:54:54 widow-mds3 kernel: Lustre: Skipped 74 previous similar messages Jun 6 09:54:54 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client 6ef35e78-c013-6641-a9a9-7c547a3db5b5 (at 5205@ptl1) in 1202 seconds. I think it's dead, and I am evicting it. Jun 6 09:54:54 widow-mds3 kernel: Lustre: Skipped 2883 previous similar messages Jun 6 09:54:55 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client e430931b-0792-4e57-f07b-92b532180789 (at 20767@ptl1) in 1100 seconds. I think it's dead, and I am evicting it. Jun 6 09:54:55 widow-mds3 kernel: Lustre: Skipped 4885 previous similar messages Jun 6 10:01:28 widow-mds3 kernel: LustreError: 12831:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-13194@ptl1 Jun 6 10:01:28 widow-mds3 kernel: LustreError: 12831:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810d9da63400 x927311/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307369488 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:01:30 widow-mds3 kernel: LustreError: 11966:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-9612@ptl1 Jun 6 10:01:30 widow-mds3 kernel: LustreError: 11966:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810e6e73e800 x964630/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307369490 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:01:44 widow-mds3 kernel: LustreError: 11777:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-3400@ptl1 Jun 6 10:01:44 widow-mds3 kernel: LustreError: 11777:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810a461c1800 x933378/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307369504 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:02:22 widow-mds3 kernel: LustreError: 11716:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-15@ptl1 Jun 6 10:02:22 widow-mds3 kernel: LustreError: 11716:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8100956f8c00 x3659997/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307369542 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:02:25 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client e646800d-0136-3cbd-1dd4-3972dc631ce6 (at 13947@ptl1) in 1259 seconds. I think it's dead, and I am evicting it. Jun 6 10:02:25 widow-mds3 kernel: Lustre: Skipped 10828 previous similar messages Jun 6 10:02:35 widow-mds3 kernel: LustreError: 11677:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-17935@ptl1 Jun 6 10:02:35 widow-mds3 kernel: LustreError: 11677:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff81016886cc00 x2790153/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307369555 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:02:43 widow-mds3 kernel: LustreError: 12699:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-20@ptl1 Jun 6 10:02:43 widow-mds3 kernel: LustreError: 12699:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810e29b06800 x16890875/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307369563 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:03:08 widow-mds3 kernel: LustreError: 12453:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-16@ptl1 Jun 6 10:03:08 widow-mds3 kernel: LustreError: 12453:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff811000464000 x901825/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307369588 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:03:40 widow-mds3 kernel: LustreError: 11238:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-8@ptl1 Jun 6 10:03:40 widow-mds3 kernel: LustreError: 11238:0:(handler.c:1514:mds_handle()) Skipped 1 previous similar message Jun 6 10:03:40 widow-mds3 kernel: LustreError: 11238:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810e77ed5450 x33587480/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307369620 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:03:40 widow-mds3 kernel: LustreError: 11238:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Jun 6 10:05:30 widow-mds3 kernel: LustreError: 11205:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-1895@ptl1 Jun 6 10:05:30 widow-mds3 kernel: LustreError: 11205:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810cd0c3e800 x899027/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307369730 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:07:46 widow-mds3 kernel: LustreError: 11652:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-11926@ptl1 Jun 6 10:07:46 widow-mds3 kernel: LustreError: 11652:0:(handler.c:1514:mds_handle()) Skipped 4 previous similar messages Jun 6 10:07:46 widow-mds3 kernel: LustreError: 11652:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810b4bbfb000 x1430565/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307369866 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:07:46 widow-mds3 kernel: LustreError: 11652:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages Jun 6 10:10:27 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client 7a592f77-46be-ecc5-e206-f284173f7bce (at 1510@ptl1) in 1352 seconds. I think it's dead, and I am evicting it. Jun 6 10:10:27 widow-mds3 kernel: Lustre: Skipped 7 previous similar messages Jun 6 10:21:30 widow-mds3 kernel: LustreError: 11573:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-11626@ptl1 Jun 6 10:21:30 widow-mds3 kernel: LustreError: 11573:0:(handler.c:1514:mds_handle()) Skipped 3 previous similar messages Jun 6 10:21:30 widow-mds3 kernel: LustreError: 11573:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8102d6f26400 x1011771/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307370690 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:21:30 widow-mds3 kernel: LustreError: 11573:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages Jun 6 10:22:24 widow-mds3 kernel: LustreError: 12353:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-17943@ptl1 Jun 6 10:22:24 widow-mds3 kernel: LustreError: 12353:0:(handler.c:1514:mds_handle()) Skipped 17 previous similar messages Jun 6 10:22:24 widow-mds3 kernel: LustreError: 12353:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810ed8a26c00 x4656852/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307370744 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:22:24 widow-mds3 kernel: LustreError: 12353:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 17 previous similar messages Jun 6 10:22:59 widow-mds3 kernel: LustreError: 12038:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-17936@ptl1 Jun 6 10:22:59 widow-mds3 kernel: LustreError: 12038:0:(handler.c:1514:mds_handle()) Skipped 20 previous similar messages Jun 6 10:22:59 widow-mds3 kernel: LustreError: 12038:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810ee59bb400 x2878570/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307370779 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:22:59 widow-mds3 kernel: LustreError: 12038:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 20 previous similar messages Jun 6 10:24:02 widow-mds3 kernel: LustreError: 6037:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway Jun 6 10:24:02 widow-mds3 kernel: LustreError: 6037:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Skipped 7 previous similar messages Jun 6 10:24:02 widow-mds3 kernel: LustreError: 6037:0:(ldlm_request.c:1599:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Jun 6 10:24:02 widow-mds3 kernel: LustreError: 6037:0:(ldlm_request.c:1599:ldlm_cli_cancel_list()) Skipped 7 previous similar messages Jun 6 10:24:02 widow-mds3 kernel: LustreError: 6037:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway Jun 6 10:24:02 widow-mds3 kernel: LustreError: 6037:0:(ldlm_request.c:1599:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Jun 6 10:24:03 widow-mds3 kernel: Lustre: client ffff810cef80fc00 umount complete Jun 6 10:25:34 widow-mds3 kernel: LustreError: 11571:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-2415@ptl1 Jun 6 10:25:34 widow-mds3 kernel: LustreError: 11571:0:(handler.c:1514:mds_handle()) Skipped 12 previous similar messages Jun 6 10:25:34 widow-mds3 kernel: LustreError: 11571:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8104bb894000 x5125617/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307370934 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:25:34 widow-mds3 kernel: LustreError: 11571:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 12 previous similar messages Jun 6 10:29:31 widow-mds3 kernel: LustreError: 12925:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-6620@ptl1 Jun 6 10:29:31 widow-mds3 kernel: LustreError: 12925:0:(handler.c:1514:mds_handle()) Skipped 15 previous similar messages Jun 6 10:29:31 widow-mds3 kernel: LustreError: 12925:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8109bf96a800 x1003015/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307371171 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 10:29:31 widow-mds3 kernel: LustreError: 12925:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 15 previous similar messages Jun 6 11:23:36 widow-mds3 kernel: LustreError: 12768:0:(handler.c:1514:mds_handle()) operation 41 on unconnected MDS from 12345-10.36.207.186@o2ib Jun 6 11:23:36 widow-mds3 kernel: LustreError: 12768:0:(handler.c:1514:mds_handle()) Skipped 331 previous similar messages Jun 6 11:23:36 widow-mds3 kernel: LustreError: 12768:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810aa9dcec00 x1370810345531003/t0 o41-><?>@<?>:0/0 lens 192/0 e 0 to 0 dl 1307374133 ref 1 fl Interpret:/0/0 rc -107/0 Jun 6 11:23:36 widow-mds3 kernel: LustreError: 12768:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 331 previous similar messages Jun 6 11:46:08 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client b69c7c09-427a-a318-3d59-1363f4fcd710 (at 10.36.207.186@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. Jun 6 11:46:08 widow-mds3 kernel: Lustre: Skipped 3 previous similar messages Jun 6 12:16:14 widow-mds3 kernel: LustreError: 11176:0:(handler.c:1514:mds_handle()) operation 41 on unconnected MDS from 12345-10.36.207.168@o2ib Jun 6 12:16:14 widow-mds3 kernel: LustreError: 11176:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810f88b6d850 x1370508582214391/t0 o41-><?>@<?>:0/0 lens 192/0 e 0 to 0 dl 1307377291 ref 1 fl Interpret:/0/0 rc -107/0 Jun 6 12:41:55 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client 3e024876-d7aa-1c13-7572-2c2202a9cfc0 (at 10.36.207.168@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. Jun 6 16:06:42 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client 03ce7244-1b8d-9b41-2a0b-01ac87e5b4cc (at 10.36.207.168@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. Jun 6 16:37:34 widow-mds3 kernel: Lustre: 12538:0:(ldlm_lib.c:574:target_handle_reconnect()) widow3-MDT0000: 8394c81c-3549-79c7-9a87-e7ff5d92f8d0 reconnecting Jun 6 16:37:34 widow-mds3 kernel: Lustre: 12538:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 7 previous similar messages Jun 6 16:56:33 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client 6b70f72c-e7d7-70d6-6b45-9dbec493d0af (at 10.36.207.168@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. Jun 6 18:05:25 widow-mds3 kernel: Lustre: 12078:0:(ldlm_lib.c:574:target_handle_reconnect()) widow3-MDT0000: 07b800ee-24f2-25be-db14-ef532a1b0326 reconnecting Jun 6 18:10:11 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2912:kiblnd_check_txs()) Timed out tx: tx_queue, 16 seconds Jun 6 18:10:11 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2975:kiblnd_check_conns()) Timed out RDMA with 10.36.229.198@o2ib200 (21) Jun 6 18:10:36 widow-mds3 kernel: Lustre: 12820:0:(ldlm_lib.c:574:target_handle_reconnect()) widow3-MDT0000: 13cf1efc-388b-8432-5c63-0076c1b14496 reconnecting Jun 6 18:10:48 widow-mds3 kernel: Lustre: 11192:0:(ldlm_lib.c:574:target_handle_reconnect()) widow3-MDT0000: a95fbe63-c077-4fe8-561b-b80a76b66e93 reconnecting Jun 6 18:10:59 widow-mds3 kernel: Lustre: 12367:0:(ldlm_lib.c:574:target_handle_reconnect()) widow3-MDT0000: 437ecc9f-f47c-6cec-bb0f-6a2c56dc1fe3 reconnecting Jun 6 18:11:02 widow-mds3 kernel: Lustre: 11273:0:(ldlm_lib.c:574:target_handle_reconnect()) widow3-MDT0000: 6ccf643b-401e-20ee-a37c-05d66ebd1258 reconnecting Jun 6 18:11:02 widow-mds3 kernel: Lustre: 11273:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Jun 6 18:11:21 widow-mds3 kernel: Lustre: 11568:0:(ldlm_lib.c:574:target_handle_reconnect()) widow3-MDT0000: 7184fbc9-0a11-3efc-09c5-928be94a6963 reconnecting Jun 6 18:11:29 widow-mds3 kernel: Lustre: 12890:0:(ldlm_lib.c:574:target_handle_reconnect()) widow3-MDT0000: da45b7f5-f3ca-54fa-1d8b-85054f6f42c2 reconnecting Jun 6 18:11:29 widow-mds3 kernel: Lustre: 12890:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 6 previous similar messages Jun 6 18:12:17 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2912:kiblnd_check_txs()) Timed out tx: tx_queue, 15 seconds Jun 6 18:12:17 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2975:kiblnd_check_conns()) Timed out RDMA with 10.36.229.197@o2ib200 (116) Jun 6 18:13:08 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2912:kiblnd_check_txs()) Timed out tx: tx_queue, 24 seconds Jun 6 18:13:08 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2975:kiblnd_check_conns()) Timed out RDMA with 10.36.229.198@o2ib200 (21) Jun 6 18:13:15 widow-mds3 kernel: Lustre: 11453:0:(ldlm_lib.c:574:target_handle_reconnect()) widow3-MDT0000: 79a1c433-e58f-5673-da5a-4dff592f67b0 reconnecting Jun 6 18:13:15 widow-mds3 kernel: Lustre: 11453:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Jun 6 18:13:58 widow-mds3 kernel: Lustre: 11893:0:(ldlm_lib.c:574:target_handle_reconnect()) widow3-MDT0000: 4b600fef-dfea-90b0-4060-92277f1cf5de reconnecting Jun 6 18:13:58 widow-mds3 kernel: Lustre: 11893:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Jun 6 18:15:14 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2912:kiblnd_check_txs()) Timed out tx: tx_queue, 19 seconds Jun 6 18:15:14 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2975:kiblnd_check_conns()) Timed out RDMA with 10.36.229.197@o2ib200 (71) Jun 6 18:15:21 widow-mds3 kernel: Lustre: 11513:0:(ldlm_lib.c:574:target_handle_reconnect()) widow3-MDT0000: 32e7aeef-773c-b79f-7be5-7300fe7b1aa2 reconnecting Jun 6 18:15:21 widow-mds3 kernel: Lustre: 11513:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message Jun 6 18:15:31 widow-mds3 kernel: LustreError: 0:0:(ldlm_lockd.c:308:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 3132@ptl1 ns: mds-widow3-MDT0000_UUID lock: ffff8109dfe7e200/0x89ba3a5d48c0943c lrc: 3/0,0 mode: CR/CR res: 498639383/3797965955 bits 0x3 hash 0 rrc: 2 type: IBT flags: 0x4000020 remote: 0x17cc763652a8fad6 expref: 77 pid: 12382 timeout: 6052503797 Jun 6 18:15:31 widow-mds3 kernel: LustreError: 0:0:(ldlm_lockd.c:308:waiting_locks_callback()) Skipped 1 previous similar message Jun 6 18:16:04 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2912:kiblnd_check_txs()) Timed out tx: tx_queue, 24 seconds Jun 6 18:16:04 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2975:kiblnd_check_conns()) Timed out RDMA with 10.36.229.198@o2ib200 (20) Jun 6 18:16:25 widow-mds3 kernel: LustreError: 12597:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-3132@ptl1 Jun 6 18:16:25 widow-mds3 kernel: LustreError: 12597:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810879372000 x296165/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307399185 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 18:17:20 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2912:kiblnd_check_txs()) Timed out tx: tx_queue, 21 seconds Jun 6 18:17:20 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2975:kiblnd_check_conns()) Timed out RDMA with 10.36.229.197@o2ib200 (20) Jun 6 18:17:34 widow-mds3 kernel: Lustre: 12205:0:(ldlm_lib.c:574:target_handle_reconnect()) widow3-MDT0000: 206e3f30-fb8e-d230-abb7-da4b26351471 reconnecting Jun 6 18:17:34 widow-mds3 kernel: Lustre: 12205:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 197 previous similar messages Jun 6 18:18:15 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client 3fcddac6-0fc1-d8ce-ff73-885c10276593 (at 13951@ptl1) in 1352 seconds. I think it's dead, and I am evicting it. Jun 6 18:18:15 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client 723b54a1-fa4d-b00f-27ed-fa91d96bb3c6 (at 20643@ptl1) in 1176 seconds. I think it's dead, and I am evicting it. Jun 6 18:18:15 widow-mds3 kernel: Lustre: Skipped 30 previous similar messages Jun 6 18:18:47 widow-mds3 kernel: LustreError: 11714:0:(service.c:653:ptlrpc_check_req()) @@@ DROPPING req from old connection 1 < 2 req@ffff810b7b493c00 x336621/t0 o400->31a12963-e758-2341-6aea-82b5e4102ffe@NET_0x4000100004049_UUID:0/0 lens 128/0 e 0 to 0 dl 0 ref 2 fl New:/0/0 rc 0/0 Jun 6 18:18:47 widow-mds3 kernel: LustreError: 11714:0:(service.c:653:ptlrpc_check_req()) Skipped 1 previous similar message Jun 6 18:18:57 widow-mds3 kernel: LustreError: 11461:0:(service.c:653:ptlrpc_check_req()) @@@ DROPPING req from old connection 1 < 2 req@ffff8103e854c800 x215624/t0 o400->a53c8c95-e3ba-778a-8594-dd479e0cc5bc@NET_0x4000100001322_UUID:0/0 lens 128/0 e 0 to 0 dl 0 ref 2 fl New:/0/0 rc 0/0 Jun 6 18:18:59 widow-mds3 kernel: Lustre: 11398:0:(ldlm_lib.c:874:target_handle_connect()) widow3-MDT0000: refuse reconnection from a22c9d60-c551-967e-6dee-5d5108039102@4605@ptl1 to 0xffff810f1d8c6600; still busy with 1 active RPCs Jun 6 18:18:59 widow-mds3 kernel: Lustre: 11398:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 19 previous similar messages Jun 6 18:18:59 widow-mds3 kernel: LustreError: 11398:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81044ed41800 x237405/t0 o38->a22c9d60-c551-967e-6dee-5d5108039102@NET_0x40001000011fd_UUID:0/0 lens 304/200 e 0 to 0 dl 1307399339 ref 1 fl Interpret:/0/0 rc -16/0 Jun 6 18:18:59 widow-mds3 kernel: LustreError: 11398:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages Jun 6 18:18:59 widow-mds3 kernel: LustreError: 11398:0:(service.c:653:ptlrpc_check_req()) @@@ DROPPING req from old connection 1 < 2 req@ffff8109723ff000 x206948/t0 o400->29066e02-107d-718b-35fd-2ba402c3e388@NET_0x4000100000446_UUID:0/0 lens 128/0 e 0 to 0 dl 0 ref 2 fl New:/0/0 rc 0/0 Jun 6 18:18:59 widow-mds3 kernel: LustreError: 11398:0:(service.c:653:ptlrpc_check_req()) Skipped 5 previous similar messages Jun 6 18:19:01 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2912:kiblnd_check_txs()) Timed out tx: tx_queue, 0 seconds Jun 6 18:19:01 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2975:kiblnd_check_conns()) Timed out RDMA with 10.36.229.197@o2ib200 (100) Jun 6 18:19:03 widow-mds3 kernel: Lustre: 12087:0:(ldlm_lib.c:874:target_handle_connect()) widow3-MDT0000: refuse reconnection from 6e536b04-8cc2-2325-db58-465577d9cd6d@17932@ptl1 to 0xffff810f21f13200; still busy with 1 active RPCs Jun 6 18:19:03 widow-mds3 kernel: LustreError: 12087:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810880cc4800 x3170089/t0 o38->6e536b04-8cc2-2325-db58-465577d9cd6d@NET_0x400010000460c_UUID:0/0 lens 304/200 e 0 to 0 dl 1307399343 ref 1 fl Interpret:/0/0 rc -16/0 Jun 6 18:19:04 widow-mds3 kernel: LustreError: 11146:0:(service.c:653:ptlrpc_check_req()) @@@ DROPPING req from old connection 1 < 2 req@ffff8105ee43c800 x217464/t0 o400->6da60279-a4eb-b77b-649f-c999b23df86e@NET_0x400010000303e_UUID:0/0 lens 128/0 e 0 to 0 dl 0 ref 2 fl New:/0/0 rc 0/0 Jun 6 18:19:04 widow-mds3 kernel: LustreError: 11146:0:(service.c:653:ptlrpc_check_req()) Skipped 12 previous similar messages Jun 6 18:19:12 widow-mds3 kernel: LustreError: 12091:0:(service.c:653:ptlrpc_check_req()) @@@ DROPPING req from old connection 1 < 2 req@ffff8104a8750400 x192841/t0 o400->4e7a18e8-9e38-451a-2d9d-bb94ba762689@NET_0x40001000013be_UUID:0/0 lens 128/0 e 0 to 0 dl 0 ref 2 fl New:/0/0 rc 0/0 Jun 6 18:19:12 widow-mds3 kernel: LustreError: 12091:0:(service.c:653:ptlrpc_check_req()) Skipped 1 previous similar message Jun 6 18:20:26 widow-mds3 kernel: Lustre: 11935:0:(ldlm_lib.c:874:target_handle_connect()) widow3-MDT0000: refuse reconnection from 9cb604dc-ec22-4ef8-a6f6-ecb35e8f1e2b@11520@ptl1 to 0xffff810f6e59dc00; still busy with 1 active RPCs Jun 6 18:20:26 widow-mds3 kernel: LustreError: 11935:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c65f20c00 x214497/t0 o38->9cb604dc-ec22-4ef8-a6f6-ecb35e8f1e2b@NET_0x4000100002d00_UUID:0/0 lens 304/200 e 0 to 0 dl 1307399426 ref 1 fl Interpret:/0/0 rc -16/0 Jun 6 18:20:42 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2912:kiblnd_check_txs()) Timed out tx: tx_queue, 0 seconds Jun 6 18:20:42 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2912:kiblnd_check_txs()) Skipped 1 previous similar message Jun 6 18:20:42 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2975:kiblnd_check_conns()) Timed out RDMA with 10.36.229.197@o2ib200 (100) Jun 6 18:20:42 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2975:kiblnd_check_conns()) Skipped 1 previous similar message Jun 6 18:21:14 widow-mds3 kernel: Lustre: 11637:0:(ldlm_lib.c:874:target_handle_connect()) widow3-MDT0000: refuse reconnection from a6f5e354-f34e-fd53-906a-54bd00115b16@4291@ptl1 to 0xffff810ef88aaa00; still busy with 1 active RPCs Jun 6 18:21:14 widow-mds3 kernel: LustreError: 11637:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8108583f9400 x206190/t0 o38->a6f5e354-f34e-fd53-906a-54bd00115b16@NET_0x40001000010c3_UUID:0/0 lens 304/200 e 0 to 0 dl 1307399474 ref 1 fl Interpret:/0/0 rc -16/0 Jun 6 18:21:17 widow-mds3 kernel: Lustre: 11994:0:(ldlm_lib.c:874:target_handle_connect()) widow3-MDT0000: refuse reconnection from edf44e70-f8d9-06ef-2506-243a72819656@393@ptl1 to 0xffff810faa96ce00; still busy with 1 active RPCs Jun 6 18:21:17 widow-mds3 kernel: Lustre: 11994:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 1 previous similar message Jun 6 18:21:17 widow-mds3 kernel: LustreError: 12543:0:(service.c:653:ptlrpc_check_req()) @@@ DROPPING req from old connection 1 < 2 req@ffff8108d6251400 x219825/t0 o400->ab9cd00d-6558-17a3-e8bd-4a65dd7d6111@NET_0x40001000052fc_UUID:0/0 lens 128/0 e 0 to 0 dl 0 ref 2 fl New:/0/0 rc 0/0 Jun 6 18:21:17 widow-mds3 kernel: LustreError: 12543:0:(service.c:653:ptlrpc_check_req()) Skipped 7 previous similar messages Jun 6 18:21:21 widow-mds3 kernel: Lustre: 11093:0:(ldlm_lib.c:874:target_handle_connect()) widow3-MDT0000: refuse reconnection from 5d0e9d5a-e0a7-10cd-2bc1-eea976a0df52@9165@ptl1 to 0xffff810f14d92600; still busy with 2 active RPCs Jun 6 18:21:21 widow-mds3 kernel: Lustre: 11093:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 27 previous similar messages Jun 6 18:21:22 widow-mds3 kernel: LustreError: 11859:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8108f8ed2c00 x212134/t0 o38->497025ca-bc5d-1e49-9b7d-4ecd2746b69b@NET_0x4000100003d04_UUID:0/0 lens 304/200 e 0 to 0 dl 1307399482 ref 1 fl Interpret:/0/0 rc -16/0 Jun 6 18:21:22 widow-mds3 kernel: LustreError: 11859:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 38 previous similar messages Jun 6 18:21:30 widow-mds3 kernel: Lustre: 12414:0:(ldlm_lib.c:874:target_handle_connect()) widow3-MDT0000: refuse reconnection from 1dd0c5c4-a41f-da29-fb79-9be60a492643@11544@ptl1 to 0xffff810fab4ebe00; still busy with 1 active RPCs Jun 6 18:21:30 widow-mds3 kernel: Lustre: 12414:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 53 previous similar messages Jun 6 18:21:50 widow-mds3 kernel: Lustre: 12779:0:(ldlm_lib.c:574:target_handle_reconnect()) widow3-MDT0000: 3890ebf2-51ff-57fb-7ce2-b86f6105acc0 reconnecting Jun 6 18:21:50 widow-mds3 kernel: Lustre: 12779:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 6404 previous similar messages Jun 6 18:21:58 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2912:kiblnd_check_txs()) Timed out tx: active_txs, 23 seconds Jun 6 18:21:58 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2975:kiblnd_check_conns()) Timed out RDMA with 10.36.229.198@o2ib200 (21) Jun 6 18:22:56 widow-mds3 kernel: Lustre: 11136:0:(ldlm_lib.c:874:target_handle_connect()) widow3-MDT0000: refuse reconnection from 9cb604dc-ec22-4ef8-a6f6-ecb35e8f1e2b@11520@ptl1 to 0xffff810f6e59dc00; still busy with 1 active RPCs Jun 6 18:22:56 widow-mds3 kernel: Lustre: 11136:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 2 previous similar messages Jun 6 18:22:56 widow-mds3 kernel: LustreError: 11136:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8104abd47000 x215907/t0 o38->9cb604dc-ec22-4ef8-a6f6-ecb35e8f1e2b@NET_0x4000100002d00_UUID:0/0 lens 304/200 e 0 to 0 dl 1307399576 ref 1 fl Interpret:/0/0 rc -16/0 Jun 6 18:22:56 widow-mds3 kernel: LustreError: 11136:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 47 previous similar messages Jun 6 18:23:39 widow-mds3 kernel: Lustre: 12293:0:(ldlm_lib.c:874:target_handle_connect()) widow3-MDT0000: refuse reconnection from ea65572b-db5c-1255-1c83-7dcacf186bb8@20756@ptl1 to 0xffff810faa01be00; still busy with 1 active RPCs Jun 6 18:23:39 widow-mds3 kernel: LustreError: 12293:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810d63721000 x217304/t0 o38->ea65572b-db5c-1255-1c83-7dcacf186bb8@NET_0x4000100005114_UUID:0/0 lens 304/200 e 0 to 0 dl 1307399619 ref 1 fl Interpret:/0/0 rc -16/0 Jun 6 18:23:51 widow-mds3 kernel: LustreError: 11756:0:(service.c:653:ptlrpc_check_req()) @@@ DROPPING req from old connection 1 < 2 req@ffff810e5179fc00 x251252/t0 o400->94717c24-7c10-47cb-c96c-edd1f4204aa2@NET_0x400010000481d_UUID:0/0 lens 128/0 e 0 to 0 dl 0 ref 2 fl New:/0/0 rc 0/0 Jun 6 18:23:51 widow-mds3 kernel: LustreError: 11756:0:(service.c:653:ptlrpc_check_req()) Skipped 21 previous similar messages Jun 6 18:24:52 widow-mds3 kernel: LustreError: 11176:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-13950@ptl1 Jun 6 18:24:52 widow-mds3 kernel: LustreError: 11176:0:(handler.c:1514:mds_handle()) Skipped 2 previous similar messages Jun 6 18:24:52 widow-mds3 kernel: LustreError: 11176:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810bec2ea800 x202828/t0 o400-><?>@<?>:0/0 lens 128/0 e 0 to 0 dl 1307399692 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 6 18:24:52 widow-mds3 kernel: LustreError: 11176:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 93 previous similar messages Jun 6 18:24:53 widow-mds3 kernel: LustreError: 11261:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-13938@ptl1 Jun 6 18:25:05 widow-mds3 kernel: LustreError: 12954:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-13942@ptl1 Jun 6 18:25:11 widow-mds3 kernel: LustreError: 12246:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-13992@ptl1 Jun 6 18:25:11 widow-mds3 kernel: LustreError: 12512:0:(service.c:653:ptlrpc_check_req()) @@@ DROPPING req from old connection 1 < 2 req@ffff810d63721000 x176050/t0 o400->037ad39e-0a6d-d7c0-1a61-a8a981723827@NET_0x400010000077d_UUID:0/0 lens 128/0 e 0 to 0 dl 0 ref 2 fl New:/0/0 rc 0/0 Jun 6 18:25:11 widow-mds3 kernel: LustreError: 12512:0:(service.c:653:ptlrpc_check_req()) Skipped 533 previous similar messages Jun 6 18:25:23 widow-mds3 kernel: LustreError: 12492:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-5027@ptl1 Jun 6 18:25:23 widow-mds3 kernel: LustreError: 12492:0:(handler.c:1514:mds_handle()) Skipped 9 previous similar messages Jun 6 18:25:27 widow-mds3 kernel: LustreError: 4798:0:(ldlm_lockd.c:1873:ldlm_cancel_handler()) ldlm_cancel from 3132@ptl1 arrived at 1307399127 with bad export cookie 9924308900716889328 Jun 6 18:26:10 widow-mds3 kernel: LustreError: 13107:0:(handler.c:1514:mds_handle()) operation 400 on unconnected MDS from 12345-13936@ptl1 Jun 6 18:26:10 widow-mds3 kernel: LustreError: 13107:0:(handler.c:1514:mds_handle()) Skipped 7 previous similar messages Jun 6 18:26:10 widow-mds3 kernel: Lustre: 12235:0:(ldlm_lib.c:874:target_handle_connect()) widow3-MDT0000: refuse reconnection from ea65572b-db5c-1255-1c83-7dcacf186bb8@20756@ptl1 to 0xffff810faa01be00; still busy with 1 active RPCs Jun 6 18:26:10 widow-mds3 kernel: Lustre: 12235:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 93 previous similar messages Jun 6 18:26:19 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client d76c5c1c-b6f5-d43a-78a0-380114bdc414 (at 14010@ptl1) in 1290 seconds. I think it's dead, and I am evicting it. Jun 6 18:26:19 widow-mds3 kernel: Lustre: Skipped 57 previous similar messages Jun 6 18:28:42 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2912:kiblnd_check_txs()) Timed out tx: tx_queue, 15 seconds Jun 6 18:28:42 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2912:kiblnd_check_txs()) Skipped 1 previous similar message Jun 6 18:28:42 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2975:kiblnd_check_conns()) Timed out RDMA with 10.36.229.197@o2ib200 (115) Jun 6 18:28:42 widow-mds3 kernel: LustreError: 10536:0:(o2iblnd_cb.c:2975:kiblnd_check_conns()) Skipped 1 previous similar message Jun 6 18:28:58 widow-mds3 kernel: LustreError: 0:0:(ldlm_lockd.c:308:waiting_locks_callback()) ### lock callback timer expired after 565s: evicting client at 9977@ptl1 ns: mds-widow3-MDT0000_UUID lock: ffff810f2b7e6400/0x89ba3a5d390e940b lrc: 3/0,0 mode: CR/CR res: 498076060/3797532407 bits 0x3 hash 0 rrc: 498 type: IBT flags: 0x4000020 remote: 0xf0b6d03100cb74f4 expref: 16 pid: 11447 timeout: 6053310658 Jun 6 18:29:28 widow-mds3 kernel: Lustre: 12502:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Jun 6 18:29:28 widow-mds3 kernel: req@ffff810279efc000 x215701/t0 o101->e39af796-0236-b20c-6460-3c31fccf4c3c@NET_0x4000100000480_UUID:0/0 lens 440/4304 e 0 to 0 dl 1307399373 ref 2 fl Interpret:/0/0 rc 0/0 Jun 6 18:31:09 widow-mds3 kernel: Lustre: No route to 12345-17931@ptl1 via 10.36.227.213@o2ib200 (all routers down) Jun 6 18:31:09 widow-mds3 kernel: Lustre: Skipped 15 previous similar messages Jun 6 18:31:09 widow-mds3 kernel: Lustre: 11172:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1369067238528567 sent from widow3-MDT0000 to NID 17931@ptl1 0s ago has failed due to network error (567s prior to deadline). Jun 6 18:31:09 widow-mds3 kernel: req@ffff81037b6ab800 x1369067238528567/t0 o104->@NET_0x400010000460b_UUID:15/16 lens 296/384 e 0 to 1 dl 1307400036 ref 1 fl Rpc:N/0/0 rc 0/0 Jun 6 18:31:09 widow-mds3 kernel: LustreError: 11172:0:(ldlm_lockd.c:607:ldlm_handle_ast_error()) ### client (nid 17931@ptl1) returned 0 from blocking AST ns: mds-widow3-MDT0000_UUID lock: ffff810938857e00/0x89ba3a5d49187af7 lrc: 4/0,0 mode: CR/CR res: 492569061/2063453048 bits 0x3 hash 0 rrc: 10 type: IBT flags: 0x4000020 remote: 0xf20ee4c91d62c7ca expref: 202 pid: 11310 timeout: 6054042590 Jun 6 18:35:12 widow-mds3 kernel: Lustre: 12853:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Jun 6 18:35:12 widow-mds3 kernel: req@ffff810f40f6b000 x213292/t0 o101->9af84319-c5b7-2962-b3ab-2ef7ff7c0b11@NET_0x4000100005101_UUID:0/0 lens 512/4304 e 0 to 0 dl 1307399717 ref 2 fl Interpret:/0/0 rc 0/0 Jun 6 18:35:12 widow-mds3 kernel: Lustre: 12853:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 294 previous similar messages Jun 6 18:36:16 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client 8af9f6a9-2556-9997-a00d-673434a98fda (at 13958@ptl1) in 1352 seconds. I think it's dead, and I am evicting it. Jun 6 18:36:16 widow-mds3 kernel: Lustre: Skipped 20 previous similar messages Jun 6 18:38:21 widow-mds3 kernel: LustreError: 11410:0:(ldlm_request.c:83:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1307398773, 1128s ago); not entering recovery in server code, just going back to sleep ns: mds-widow3-MDT0000_UUID lock: ffff810517df4600/0x89ba3a5d4900f9dd lrc: 3/1,0 mode: --/CR res: 498076060/3797532407 bits 0x3 hash 0 rrc: 494 type: IBT flags: 0x4004030 remote: 0x0 expref: -99 pid: 11410 timeout: 0 Jun 6 18:38:21 widow-mds3 kernel: LustreError: 11201:0:(ldlm_request.c:83:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1307398773, 1128s ago); not entering recovery in server code, just going back to sleep ns: mds-widow3-MDT0000_UUID lock: ffff810d90ed1a00/0x89ba3a5d4900f989 lrc: 3/0,1 mode: --/EX res: 498076060/3797532407 bits 0x2 hash 5e5b4c80 rrc: 494 type: IBT flags: 0x4004030 remote: 0x0 expref: -99 pid: 11201 timeout: 0 Jun 6 18:38:21 widow-mds3 kernel: LustreError: 11201:0:(ldlm_request.c:83:ldlm_expired_completion_wait()) Skipped 7 previous similar messages Jun 6 18:38:21 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399901.11956 Jun 6 18:38:21 widow-mds3 kernel: LustreError: 11239:0:(ldlm_request.c:83:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1307398773, 1128s ago); not entering recovery in server code, just going back to sleep ns: mds-widow3-MDT0000_UUID lock: ffff810614693400/0x89ba3a5d4900fb6c lrc: 3/0,1 mode: --/EX res: 498076060/3797532407 bits 0x2 hash 5e5f76d0 rrc: 494 type: IBT flags: 0x4004030 remote: 0x0 expref: -99 pid: 11239 timeout: 0 Jun 6 18:38:21 widow-mds3 kernel: LustreError: 11239:0:(ldlm_request.c:83:ldlm_expired_completion_wait()) Skipped 24 previous similar messages Jun 6 18:38:59 widow-mds3 kernel: LustreError: 0:0:(ldlm_lockd.c:308:waiting_locks_callback()) ### lock callback timer expired after 1166s: evicting client at 1161@ptl1 ns: mds-widow3-MDT0000_UUID lock: ffff810ec8b3d200/0x89ba3a5d4900f97b lrc: 3/0,0 mode: CR/CR res: 498076060/3797532407 bits 0x3 hash 0 rrc: 494 type: IBT flags: 0x4000030 remote: 0x288088ddae545782 expref: 44 pid: 12405 timeout: 6053911000 Jun 6 18:38:59 widow-mds3 kernel: LustreError: 0:0:(ldlm_lockd.c:308:waiting_locks_callback()) Skipped 1 previous similar message Jun 6 18:38:59 widow-mds3 kernel: Lustre: No route to 12345-15625@ptl1 via 10.36.227.213@o2ib200 (all routers down) Jun 6 18:38:59 widow-mds3 kernel: Lustre: Skipped 1 previous similar message Jun 6 18:38:59 widow-mds3 kernel: Lustre: 11201:0:(service.c:1434:ptlrpc_server_handle_request()) @@@ Request x201072 took longer than estimated (600+566s); client may timeout. req@ffff810ee8be2c00 x201072/t65120276798 o101->0d659513-8192-ea03-fa4b-a13a24df8c6a@NET_0x4000100003d09_UUID:0/0 lens 512/544 e 0 to 0 dl 1307399373 ref 1 fl Complete:/0/0 rc 301/301 Jun 6 18:38:59 widow-mds3 kernel: Lustre: 11201:0:(service.c:1434:ptlrpc_server_handle_request()) Skipped 1 previous similar message Jun 6 18:39:33 widow-mds3 kernel: Lustre: Service thread pid 11532 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 6 18:39:33 widow-mds3 kernel: Pid: 11532, comm: ll_mdt_466 Jun 6 18:39:33 widow-mds3 kernel: Jun 6 18:39:33 widow-mds3 kernel: Call Trace: Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f0acd>] ldlm_expired_completion_wait+0xad/0x250 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff800638b3>] schedule_timeout+0x92/0xad Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f0a20>] ldlm_expired_completion_wait+0x0/0x250 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f2562>] ldlm_completion_ast+0x4c2/0x880 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d7729>] ldlm_lock_enqueue+0x9d9/0xb20 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008d07b>] default_wake_function+0x0/0xe Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d2b8a>] ldlm_lock_addref_internal_nolock+0x3a/0x90 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f10db>] ldlm_cli_enqueue_local+0x46b/0x520 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b22267>] enqueue_ordered_locks+0x387/0x4d0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887ee9c0>] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f20a0>] ldlm_completion_ast+0x0/0x880 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b229e8>] mds_get_parent_child_locked+0x638/0x8f0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b136a2>] mds_getattr_lock+0x622/0xc80 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b0edda>] fixup_handle_for_resent_req+0x5a/0x2c0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b19dd3>] mds_intent_policy+0x623/0xc20 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d9290>] ldlm_resource_putref_internal+0x230/0x460 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d6ed6>] ldlm_lock_enqueue+0x186/0xb20 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d381d>] ldlm_lock_create+0x9bd/0x9f0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887fb8c0>] ldlm_server_blocking_ast+0x0/0x83d [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f8b89>] ldlm_handle_enqueue+0xc09/0x1210 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b18b80>] mds_handle+0x40e0/0x4d10 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff80062ff8>] thread_return+0x62/0xfe Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887006c0>] LNetMDAttach+0x370/0x4b0 [lnet] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8881ca75>] lustre_msg_get_conn_cnt+0x35/0xf0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8001cb46>] __mod_timer+0x100/0x10f Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff888263f9>] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88826b55>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008d07b>] default_wake_function+0x0/0xe Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88827ae6>] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88826b80>] ptlrpc_main+0x0/0x1120 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11 Jun 6 18:39:33 widow-mds3 kernel: Jun 6 18:39:33 widow-mds3 kernel: Pid: 12585, comm: ll_mdt_1519 Jun 6 18:39:33 widow-mds3 kernel: Jun 6 18:39:33 widow-mds3 kernel: Call Trace: Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.11532 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f0a20>] ldlm_expired_completion_wait+0x0/0x250 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f2562>] ldlm_completion_ast+0x4c2/0x880 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d7729>] ldlm_lock_enqueue+0x9d9/0xb20 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008d07b>] default_wake_function+0x0/0xe Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d2b8a>] ldlm_lock_addref_internal_nolock+0x3a/0x90 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f10db>] ldlm_cli_enqueue_local+0x46b/0x520 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b2214a>] enqueue_ordered_locks+0x26a/0x4d0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887ee9c0>] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f20a0>] ldlm_completion_ast+0x0/0x880 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b229e8>] mds_get_parent_child_locked+0x638/0x8f0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff883cfe25>] mthca_arbel_post_send+0x57d/0x58f [ib_mthca] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b44a3b>] mds_open+0xc3b/0x38fb [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff888cea54>] kiblnd_init_tx_msg+0x154/0x1d0 [ko2iblnd] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff888181c1>] lustre_swab_buf+0x81/0x170 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8000d3ba>] dput+0x2c/0x113 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b1d0d5>] mds_reint_rec+0x365/0x550 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b47cbe>] mds_update_unpack+0x1fe/0x280 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b0feca>] mds_reint+0x35a/0x420 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b0edda>] fixup_handle_for_resent_req+0x5a/0x2c0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b19c5c>] mds_intent_policy+0x4ac/0xc20 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d9290>] ldlm_resource_putref_internal+0x230/0x460 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d6ed6>] ldlm_lock_enqueue+0x186/0xb20 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d381d>] ldlm_lock_create+0x9bd/0x9f0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887fb8c0>] ldlm_server_blocking_ast+0x0/0x83d [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f8b89>] ldlm_handle_enqueue+0xc09/0x1210 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b18b80>] mds_handle+0x40e0/0x4d10 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff800767bd>] smp_send_reschedule+0x4e/0x53 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008ca15>] enqueue_task+0x41/0x56 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008ca80>] __activate_task+0x56/0x6d Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8881ca75>] lustre_msg_get_conn_cnt+0x35/0xf0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff888263f9>] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88826b55>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008b4a5>] __wake_up_common+0x3e/0x68 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88827ae6>] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88826b80>] ptlrpc_main+0x0/0x1120 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11 Jun 6 18:39:33 widow-mds3 kernel: Jun 6 18:39:33 widow-mds3 kernel: Pid: 12960, comm: ll_mdt_1894 Jun 6 18:39:33 widow-mds3 kernel: Jun 6 18:39:33 widow-mds3 kernel: Call Trace: Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f0acd>] ldlm_expired_completion_wait+0xad/0x250 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff800638b3>] schedule_timeout+0x92/0xad Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f0a20>] ldlm_expired_completion_wait+0x0/0x250 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f2562>] ldlm_completion_ast+0x4c2/0x880 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d7729>] ldlm_lock_enqueue+0x9d9/0xb20 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008d07b>] default_wake_function+0x0/0xe Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d2b8a>] ldlm_lock_addref_internal_nolock+0x3a/0x90 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f10db>] ldlm_cli_enqueue_local+0x46b/0x520 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b2214a>] enqueue_ordered_locks+0x26a/0x4d0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887ee9c0>] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f20a0>] ldlm_completion_ast+0x0/0x880 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b229e8>] mds_get_parent_child_locked+0x638/0x8f0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff80036ce3>] __lookup_hash+0x61/0x12f Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff883cfe25>] mthca_arbel_post_send+0x57d/0x58f [ib_mthca] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b44a3b>] mds_open+0xc3b/0x38fb [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff888cea54>] kiblnd_init_tx_msg+0x154/0x1d0 [ko2iblnd] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff888181c1>] lustre_swab_buf+0x81/0x170 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8000d3ba>] dput+0x2c/0x113 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b1d0d5>] mds_reint_rec+0x365/0x550 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b47cbe>] mds_update_unpack+0x1fe/0x280 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b0feca>] mds_reint+0x35a/0x420 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b0edda>] fixup_handle_for_resent_req+0x5a/0x2c0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b19c5c>] mds_intent_policy+0x4ac/0xc20 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d9290>] ldlm_resource_putref_internal+0x230/0x460 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d6ed6>] ldlm_lock_enqueue+0x186/0xb20 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d381d>] ldlm_lock_create+0x9bd/0x9f0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887fb8c0>] ldlm_server_blocking_ast+0x0/0x83d [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f8b89>] ldlm_handle_enqueue+0xc09/0x1210 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b18b80>] mds_handle+0x40e0/0x4d10 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008ca15>] enqueue_task+0x41/0x56 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008ca80>] __activate_task+0x56/0x6d Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8881ca75>] lustre_msg_get_conn_cnt+0x35/0xf0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff888263f9>] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88826b55>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008b4a5>] __wake_up_common+0x3e/0x68 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88827ae6>] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88826b80>] ptlrpc_main+0x0/0x1120 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11 Jun 6 18:39:33 widow-mds3 kernel: Jun 6 18:39:33 widow-mds3 kernel: Lustre: Service thread pid 11911 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 6 18:39:33 widow-mds3 kernel: Lustre: Skipped 2 previous similar messages Jun 6 18:39:33 widow-mds3 kernel: Pid: 11911, comm: ll_mdt_845 Jun 6 18:39:33 widow-mds3 kernel: Jun 6 18:39:33 widow-mds3 kernel: Call Trace: Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f0acd>] ldlm_expired_completion_wait+0xad/0x250 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff800638b3>] schedule_timeout+0x92/0xad Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f0a20>] ldlm_expired_completion_wait+0x0/0x250 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f2562>] ldlm_completion_ast+0x4c2/0x880 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d7729>] ldlm_lock_enqueue+0x9d9/0xb20 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008d07b>] default_wake_function+0x0/0xe Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d2b8a>] ldlm_lock_addref_internal_nolock+0x3a/0x90 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f10db>] ldlm_cli_enqueue_local+0x46b/0x520 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b2214a>] enqueue_ordered_locks+0x26a/0x4d0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887ee9c0>] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f20a0>] ldlm_completion_ast+0x0/0x880 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b229e8>] mds_get_parent_child_locked+0x638/0x8f0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff80036ce3>] __lookup_hash+0x61/0x12f Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff883cfe25>] mthca_arbel_post_send+0x57d/0x58f [ib_mthca] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b44a3b>] mds_open+0xc3b/0x38fb [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff888cea54>] kiblnd_init_tx_msg+0x154/0x1d0 [ko2iblnd] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff888181c1>] lustre_swab_buf+0x81/0x170 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8000d3ba>] dput+0x2c/0x113 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b1d0d5>] mds_reint_rec+0x365/0x550 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b47cbe>] mds_update_unpack+0x1fe/0x280 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b0feca>] mds_reint+0x35a/0x420 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b0edda>] fixup_handle_for_resent_req+0x5a/0x2c0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b19c5c>] mds_intent_policy+0x4ac/0xc20 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d9290>] ldlm_resource_putref_internal+0x230/0x460 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d6ed6>] ldlm_lock_enqueue+0x186/0xb20 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d381d>] ldlm_lock_create+0x9bd/0x9f0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887fb8c0>] ldlm_server_blocking_ast+0x0/0x83d [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f8b89>] ldlm_handle_enqueue+0xc09/0x1210 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b18b80>] mds_handle+0x40e0/0x4d10 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008ca15>] enqueue_task+0x41/0x56 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008ca80>] __activate_task+0x56/0x6d Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8881ca75>] lustre_msg_get_conn_cnt+0x35/0xf0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff888263f9>] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88826b55>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008b4a5>] __wake_up_common+0x3e/0x68 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88827ae6>] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88826b80>] ptlrpc_main+0x0/0x1120 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11 Jun 6 18:39:33 widow-mds3 kernel: Jun 6 18:39:33 widow-mds3 kernel: Pid: 11727, comm: ll_mdt_661 Jun 6 18:39:33 widow-mds3 kernel: Jun 6 18:39:33 widow-mds3 kernel: Call Trace: Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f0acd>] ldlm_expired_completion_wait+0xad/0x250 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff800638b3>] schedule_timeout+0x92/0xad Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f0a20>] ldlm_expired_completion_wait+0x0/0x250 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f2562>] ldlm_completion_ast+0x4c2/0x880 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d7729>] ldlm_lock_enqueue+0x9d9/0xb20 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008d07b>] default_wake_function+0x0/0xe Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d2b8a>] ldlm_lock_addref_internal_nolock+0x3a/0x90 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f10db>] ldlm_cli_enqueue_local+0x46b/0x520 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b2214a>] enqueue_ordered_locks+0x26a/0x4d0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887ee9c0>] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f20a0>] ldlm_completion_ast+0x0/0x880 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b229e8>] mds_get_parent_child_locked+0x638/0x8f0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff80036ce3>] __lookup_hash+0x61/0x12f Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff883cfe25>] mthca_arbel_post_send+0x57d/0x58f [ib_mthca] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b44a3b>] mds_open+0xc3b/0x38fb [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8000f2cf>] __alloc_pages+0x78/0x308 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff80017b11>] cache_grow+0x35a/0x3c1 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff888181c1>] lustre_swab_buf+0x81/0x170 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8000d3ba>] dput+0x2c/0x113 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b1d0d5>] mds_reint_rec+0x365/0x550 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b47cbe>] mds_update_unpack+0x1fe/0x280 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b0feca>] mds_reint+0x35a/0x420 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b0edda>] fixup_handle_for_resent_req+0x5a/0x2c0 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b19c5c>] mds_intent_policy+0x4ac/0xc20 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d9290>] ldlm_resource_putref_internal+0x230/0x460 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d6ed6>] ldlm_lock_enqueue+0x186/0xb20 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887d381d>] ldlm_lock_create+0x9bd/0x9f0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887fb8c0>] ldlm_server_blocking_ast+0x0/0x83d [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff887f8b89>] ldlm_handle_enqueue+0xc09/0x1210 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88b18b80>] mds_handle+0x40e0/0x4d10 [mds] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff80062ff8>] thread_return+0x62/0xfe Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8881ca75>] lustre_msg_get_conn_cnt+0x35/0xf0 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8001cb46>] __mod_timer+0x100/0x10f Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff888263f9>] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88826b55>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8008d07b>] default_wake_function+0x0/0xe Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88827ae6>] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11 Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff88826b80>] ptlrpc_main+0x0/0x1120 [ptlrpc] Jun 6 18:39:33 widow-mds3 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11 Jun 6 18:39:33 widow-mds3 kernel: Jun 6 18:39:33 widow-mds3 kernel: Lustre: Service thread pid 12611 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jun 6 18:39:33 widow-mds3 kernel: Lustre: Service thread pid 11099 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jun 6 18:39:33 widow-mds3 kernel: Lustre: Skipped 35 previous similar messages Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.11821 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.11245 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.11089 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.11498 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.11434 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.11898 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.12760 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.12606 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.11092 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.12927 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.12732 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.11866 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.11936 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.13092 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.13113 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.12281 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.11457 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.11278 Jun 6 18:39:33 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399973.11213 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12031 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11449 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12183 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12336 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12257 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12163 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12557 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11519 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11388 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11326 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11422 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11351 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11664 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11421 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11972 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11489 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11394 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11686 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.13058 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11214 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12064 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12214 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11354 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12770 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12879 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12242 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11264 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11080 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12480 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12909 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11705 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.13105 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12071 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12019 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11841 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12288 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12656 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12854 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12527 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12436 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12674 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11942 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11611 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11340 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11332 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12526 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11244 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12767 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.13097 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.13108 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12859 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12124 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11440 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11968 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12546 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11814 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12046 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.13072 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12259 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12303 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12912 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12006 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11411 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12171 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12213 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11682 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11572 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12874 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.13002 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12204 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12885 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12684 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11615 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11634 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12548 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12043 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12150 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12614 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12037 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12790 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12488 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12174 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12675 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.12746 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.13094 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11640 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11582 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11977 Jun 6 18:39:34 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.11115 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399974.13089 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12881 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11757 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11315 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12523 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11260 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12386 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11742 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12126 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12223 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12097 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12144 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12980 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11683 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12868 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11878 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11676 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12858 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11860 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12313 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.13065 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11508 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11328 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11736 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12490 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11723 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11067 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12317 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12120 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12209 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11451 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12002 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11377 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11934 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12089 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12331 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12491 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.13068 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.13025 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12809 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12207 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12498 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11926 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11917 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11175 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12247 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12639 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11311 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11959 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11781 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11893 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11393 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11939 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.13077 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12425 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12594 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12069 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11515 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11982 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11277 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12038 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11471 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11419 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12774 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11672 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.13011 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12250 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11861 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11429 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12542 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11495 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12095 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12972 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11248 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11287 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12658 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12637 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12285 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12895 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12515 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.13054 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.13066 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11119 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12410 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12429 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11886 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12081 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12178 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.13016 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.11638 Jun 6 18:39:35 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399975.12911 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12914 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12556 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11523 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12953 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12346 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12279 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12466 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12005 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12736 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12906 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12707 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11486 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11467 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11472 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12800 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11583 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11460 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12109 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12522 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12864 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11655 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12119 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12862 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11850 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11202 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12636 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12381 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11133 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11304 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12016 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.13084 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11797 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11268 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11296 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11988 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.13059 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12362 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11677 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12711 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12341 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11525 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12088 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11431 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11962 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12509 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11099 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11718 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11124 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11733 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11853 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12518 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11257 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11530 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11889 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12754 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11923 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12444 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12871 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12775 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11915 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12166 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11239 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11759 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12471 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11082 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11325 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12751 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12677 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12941 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11617 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11762 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12850 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12995 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12910 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11127 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12312 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11895 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12062 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11071 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.13050 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11220 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12611 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11727 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.11911 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12960 Jun 6 18:39:36 widow-mds3 kernel: LustreError: dumping log to /tmp/lustre-log.1307399976.12585 Jun 6 18:43:49 widow-mds3 kernel: Lustre: widow3-MDT0000: haven't heard from client e20943fa-f2e8-d9ef-818e-6bf5764b2de0 (at 8613@ptl1) in 1352 seconds. I think it's dead, and I am evicting it. Jun 6 18:43:49 widow-mds3 kernel: Lustre: Skipped 3 previous similar messages Jun 6 18:43:52 widow-mds3 kernel: Lustre: 11127:0:(mds_reint.c:238:mds_finish_transno()) commit transaction for disconnected client ef8200fc-1250-77cb-d195-3179a105e0a9: rc 0 Jun 6 18:43:52 widow-mds3 kernel: Lustre: No route to 12345-10753@ptl1 via 10.36.227.213@o2ib200 (all routers down) Jun 6 18:43:52 widow-mds3 kernel: Lustre: Skipped 8 previous similar messages Jun 6 18:43:52 widow-mds3 kernel: Lustre: 11127:0:(service.c:1434:ptlrpc_server_handle_request()) @@@ Request x210349 took longer than estimated (600+859s); client may timeout. req@ffff8105f1566000 x210349/t0 o101->ef8200fc-1250-77cb-d195-3179a105e0a9@NET_0x4000100002a01_UUID:0/0 lens 512/544 e 0 to 0 dl 1307399373 ref 1 fl Complete:/0/0 rc -107/-107 Jun 6 18:43:52 widow-mds3 kernel: Lustre: 11127:0:(service.c:1434:ptlrpc_server_handle_request()) Skipped 7 previous similar messages Jun 6 18:43:52 widow-mds3 kernel: Lustre: Service thread pid 11127 completed after 1458.65s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 6 18:43:52 widow-mds3 kernel: LustreError: 12910:0:(ldlm_lockd.c:1174:ldlm_handle_enqueue()) ### lock on destroyed export ffff810f21719800 ns: mds-widow3-MDT0000_UUID lock: ffff810418428400/0x89ba3a5d4900fa93 lrc: 3/0,0 mode: CR/CR res: 498076060/3797532407 bits 0x3 hash 0 rrc: 450 type: IBT flags: 0x4000030 remote: 0xa8f625c652e30df2 expref: 3 pid: 12910 timeout: 0 Jun 6 18:43:52 widow-mds3 kernel: LustreError: 12910:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8108aa60a400 x209787/t0 o101->220d8c48-429b-bada-847a-4136636c9f90@NET_0x4000100000481_UUID:0/0 lens 440/416 e 0 to 0 dl 1307399373 ref 1 fl Interpret:/0/0 rc -107/0 Jun 6 18:43:52 widow-mds3 kernel: LustreError: 12910:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 105 previous similar messages Jun 6 18:43:52 widow-mds3 kernel: Lustre: Service thread pid 12166 completed after 1458.65s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 6 18:43:52 widow-mds3 kernel: Lustre: Skipped 15 previous similar messages Jun 6 18:43:52 widow-mds3 kernel: Lustre: 12960:0:(mds_reint.c:238:mds_finish_transno()) commit transaction for disconnected client be7451c2-56ab-552a-dce7-3d12d97bb32d: rc 0 Jun 6 18:43:52 widow-mds3 kernel: Lustre: 12960:0:(mds_reint.c:238:mds_finish_transno()) Skipped 4 previous similar messages Jun 6 18:43:52 widow-mds3 kernel: LustreError: 12775:0:(ldlm_lockd.c:1174:ldlm_handle_enqueue()) ### lock on destroyed export ffff810f1a199000 ns: mds-widow3-MDT0000_UUID lock: ffff81095f01e000/0x89ba3a5d4900fb81 lrc: 3/0,0 mode: CR/CR res: 498076060/3797532407 bits 0x3 hash 0 rrc: 431 type: IBT flags: 0x4000030 remote: 0xc90366bbe72d76d expref: 3 pid: 12775 timeout: 0 Jun 6 18:43:52 widow-mds3 kernel: LustreError: 12775:0:(ldlm_lockd.c:1174:ldlm_handle_enqueue()) Skipped 12 previous similar messages