* Sequence of injected faults: ---- SOAK CONFIG ONLY mds_restart enabled mds_restart : 2016-03-15 08:16:39,710 - 2016-03-15 08:26:51,373 lola-10 mds_restart : 2016-03-15 08:55:12,374 - 2016-03-15 09:04:09,205 lola-8 mds_restart : 2016-03-15 09:41:35,206 - 2016-03-15 09:48:12,189 lola-10 mds_restart : 2016-03-15 09:51:24,191 - 2016-03-15 10:01:44,366 lola-8 mds_restart : 2016-03-15 10:23:14,368 - 2016-03-15 10:30:32,034 lola-11 mds_restart : 2016-03-15 11:11:13,035 - 2016-03-15 11:21:35,789 lola-9 mds_restart : 2016-03-15 11:49:43,791 - 2016-03-15 11:58:15,780 lola-8 mds_restart : 2016-03-15 12:25:02,782 - 2016-03-15 12:30:55,962 lola-10 mds_restart : 2016-03-15 12:46:18,964 - 2016-03-15 12:53:54,851 lola-11 mds_restart : 2016-03-15 13:22:55,853 - 2016-03-15 13:28:35,769 lola-10 mds_restart : 2016-03-15 14:24:49,771 - 2016-03-15 14:35:17,158 lola-11 mds_restart : 2016-03-15 15:20:31,159 - 2016-03-15 15:29:13,469 lola-11 mds_restart : 2016-03-15 15:36:54,470 - 2016-03-15 15:48:31,288 lola-9 mds_restart : 2016-03-15 16:07:00,290 - 2016-03-15 16:14:32,337 lola-9 mds_restart : 2016-03-15 17:10:33,339 - 2016-03-15 17:20:12,707 lola-9 mds_restart : 2016-03-15 17:22:50,709 - 2016-03-15 17:35:09,687 lola-8 mds_restart : 2016-03-15 18:14:03,689 - 2016-03-15 18:21:59,703 lola-11 mds_restart : 2016-03-15 18:54:05,705 - 2016-03-15 19:04:01,795 lola-10 mds_restart : 2016-03-15 19:34:17,796 - 2016-03-15 19:40:20,006 lola-11 mds_restart : 2016-03-15 20:03:18,008 - 2016-03-15 20:11:23,427 lola-11 mds_restart : 2016-03-15 20:48:01,428 - 2016-03-15 20:55:37,661 lola-10 mds_restart : 2016-03-15 21:30:00,663 - 2016-03-15 21:38:40,471 lola-9 mds_restart : 2016-03-15 21:50:05,472 - 2016-03-15 22:00:07,730 lola-8 mds_restart : 2016-03-15 22:32:34,732 - 2016-03-15 22:40:45,011 lola-11 mds_restart : 2016-03-15 23:07:16,013 - 2016-03-15 23:15:26,711 lola-11 mds_restart : 2016-03-15 23:24:57,712 - 2016-03-15 23:30:43,695 lola-10 mds_restart : 2016-03-15 23:47:05,696 - 2016-03-15 23:55:19,907 lola-8 mds_restart : 2016-03-16 00:08:46,909 - 2016-03-16 00:17:41,207 lola-11 mds_restart : 2016-03-16 00:27:17,209 - 2016-03-16 00:36:00,333 lola-8 mds_restart : 2016-03-16 01:04:07,334 - 2016-03-16 01:11:10,736 lola-10 mds_restart : 2016-03-16 02:09:05,737 - 2016-03-16 02:18:46,185 lola-9 mds_restart : 2016-03-16 02:43:37,186 - 2016-03-16 02:51:18,233 lola-11 mds_restart : 2016-03-16 03:46:34,234 - 2016-03-16 03:55:43,944 lola-10 mds_restart : 2016-03-16 04:40:25,946 - 2016-03-16 04:49:18,939 lola-8 mds_restart : 2016-03-16 04:53:58,941 - 2016-03-16 05:03:22,201 lola-11 mds_restart : 2016-03-16 05:57:54,203 - 2016-03-16 06:05:30,247 lola-10 mds_restart : 2016-03-16 06:41:27,248 - 2016-03-16 06:49:59,223 lola-8 mds_restart : 2016-03-16 07:42:41,225 - 2016-03-16 07:54:35,848 lola-8 mds_restart : 2016-03-16 08:07:58,849 - 2016-03-16 08:17:08,541 lola-10 ---- SOAK CONFIG changed: enabled mds_failover + interrupt of recovery on secondary node mds_failover : 2016-03-16 09:00:08,189 - 2016-03-16 09:06:14,026 lola-11 mds_failover : 2016-03-16 09:44:56,027 - 2016-03-16 09:52:52,171 lola-8 mds_failover : 2016-03-16 10:20:15,173 - 2016-03-16 10:28:19,030 lola-8 mds_failover : 2016-03-16 11:09:50,032 - 2016-03-16 11:18:18,572 lola-8 mds_failover : 2016-03-16 11:44:33,573 - 2016-03-16 11:50:52,153 lola-10 mds_failover : 2016-03-16 11:51:20,155 - 2016-03-16 11:59:26,400 lola-9 mds_failover : 2016-03-16 12:42:08,402 - 2016-03-16 12:50:26,894 lola-8 mds_failover : 2016-03-16 12:59:14,895 - 2016-03-16 13:07:26,758 lola-9 mds_failover : 2016-03-16 14:00:46,759 - 2016-03-16 14:08:57,011 lola-8 mds_failover : 2016-03-16 14:21:26,013 - 2016-03-16 14:27:25,564 lola-11 mds_failover : 2016-03-16 15:04:14,565 - 2016-03-16 15:12:20,120 lola-8 mds_restart : 2016-03-16 16:08:31,121 - 2016-03-16 16:17:47,816 lola-11 mds_failover : 2016-03-16 17:05:24,818 - 2016-03-16 17:11:24,067 lola-10 mds_restart : 2016-03-16 17:46:36,068 - 2016-03-16 17:55:27,404 lola-11 mds_failover : 2016-03-16 17:56:27,405 - 2016-03-16 18:04:32,659 lola-9 mds_failover : 2016-03-16 18:07:35,660 - 2016-03-16 18:15:38,963 lola-8 mds_restart : 2016-03-16 19:15:12,964 - 2016-03-16 19:24:15,544 lola-10 mds_restart : 2016-03-16 19:55:20,546 - 2016-03-16 20:02:08,214 lola-10 mds_restart : 2016-03-16 20:19:27,216 - 2016-03-16 20:27:26,120 lola-9 mds_restart : 2016-03-16 20:59:09,122 - 2016-03-16 21:07:40,842 lola-10 mds_restart : 2016-03-16 22:04:28,843 - 2016-03-16 22:13:16,976 lola-9 mds_failover : 2016-03-16 22:40:28,977 - 2016-03-16 22:47:28,683 lola-10 mds_restart : 2016-03-16 22:55:56,685 - 2016-03-16 23:07:33,229 lola-9 mds_restart : 2016-03-16 23:19:34,230 - 2016-03-16 23:28:51,173 lola-10 mds_failover : 2016-03-16 23:57:50,175 - 2016-03-17 00:04:00,765 lola-11 mds_failover : 2016-03-17 00:36:40,766 - 2016-03-17 00:44:48,814 lola-8 mds_restart : 2016-03-17 01:30:32,816 - 2016-03-17 01:42:42,982 lola-8 mds_restart : 2016-03-17 01:59:58,984 - 2016-03-17 02:08:43,051 lola-8 mds_restart : 2016-03-17 02:17:02,053 - 2016-03-17 02:28:13,840 lola-9 mds_restart : 2016-03-17 03:05:45,842 - 2016-03-17 03:12:31,610 lola-11 mds_failover : 2016-03-17 03:57:40,611 - 2016-03-17 04:05:30,972 lola-9 mds_failover : 2016-03-17 04:26:14,974 - 2016-03-17 04:32:17,897 lola-10 mds_restart : 2016-03-17 05:05:48,958 - 2016-03-17 05:13:59,161 lola-10 mds_restart : 2016-03-17 06:10:59,162 - 2016-03-17 06:23:09,986 lola-9 mds_failover : 2016-03-17 06:49:08,987 - 2016-03-17 06:58:22,000 lola-9 mds_failover : 2016-03-17 07:25:41,001 - 2016-03-17 07:33:07,955 lola-10 * Recovery times Mar 15 09:05:37 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 1:23, of 12 clients 12 recovered and 0 were evicted. Mar 15 10:03:21 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 1:34, of 12 clients 12 recovered and 0 were evicted. Mar 15 11:59:28 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 1:12, of 12 clients 12 recovered and 0 were evicted. Mar 15 17:36:43 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 1:30, of 12 clients 12 recovered and 0 were evicted. Mar 15 22:01:00 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 0:33, of 12 clients 12 recovered and 0 were evicted. Mar 15 23:56:06 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 0:32, of 12 clients 12 recovered and 0 were evicted. Mar 16 00:37:22 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 1:18, of 12 clients 12 recovered and 0 were evicted. Mar 16 04:51:04 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 1:32, of 12 clients 12 recovered and 0 were evicted. Mar 16 06:50:59 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 0:57, of 12 clients 12 recovered and 0 were evicted. Mar 16 07:55:34 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 0:47, of 12 clients 12 recovered and 0 were evicted. Mar 16 09:53:48 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 0:44, of 12 clients 12 recovered and 0 were evicted. Mar 16 10:29:35 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 1:10, of 12 clients 12 recovered and 0 were evicted. Mar 16 11:18:56 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 0:32, of 12 clients 12 recovered and 0 were evicted. Mar 16 11:59:15 lola-8 kernel: Lustre: soaked-MDT0001: Recovery over after 0:14, of 12 clients 0 recovered and 12 were evicted. Mar 16 12:57:12 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 6:39, of 12 clients 12 recovered and 0 were evicted. Mar 16 13:07:16 lola-8 kernel: Lustre: soaked-MDT0001: Recovery over after 0:19, of 12 clients 0 recovered and 12 were evicted. Mar 16 14:09:49 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 0:43, of 12 clients 12 recovered and 0 were evicted. Mar 16 15:13:17 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 0:35, of 12 clients 12 recovered and 0 were evicted. Mar 16 18:04:21 lola-8 kernel: Lustre: soaked-MDT0001: Recovery over after 0:01, of 12 clients 0 recovered and 12 were evicted. Mar 16 18:16:34 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 0:48, of 12 clients 12 recovered and 0 were evicted. Mar 17 00:54:51 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 9:56, of 12 clients 12 recovered and 0 were evicted. Mar 17 01:44:28 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 1:25, of 12 clients 12 recovered and 0 were evicted. Mar 17 02:10:09 lola-8 kernel: Lustre: soaked-MDT0000: Recovery over after 1:21, of 12 clients 12 recovered and 0 were evicted. Mar 17 04:05:20 lola-8 kernel: Lustre: soaked-MDT0001: Recovery over after 0:15, of 12 clients 0 recovered and 12 were evicted. Mar 17 06:58:05 lola-8 kernel: Lustre: soaked-MDT0001: Recovery over after 0:51, of 12 clients 12 recovered and 0 were evicted. Mar 15 11:23:15 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 1:32, of 12 clients 12 recovered and 0 were evicted. Mar 15 15:49:54 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 1:18, of 12 clients 12 recovered and 0 were evicted. Mar 15 16:15:44 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 1:07, of 12 clients 12 recovered and 0 were evicted. Mar 15 17:21:54 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 1:36, of 12 clients 12 recovered and 0 were evicted. Mar 15 21:40:00 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 1:19, of 12 clients 12 recovered and 0 were evicted. Mar 16 02:19:39 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 0:52, of 12 clients 12 recovered and 0 were evicted. Mar 16 09:52:35 lola-9 kernel: Lustre: soaked-MDT0000: Recovery over after 0:15, of 12 clients 0 recovered and 12 were evicted. Mar 16 11:18:02 lola-9 kernel: Lustre: soaked-MDT0000: Recovery over after 0:08, of 12 clients 0 recovered and 12 were evicted. Mar 16 12:07:03 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 7:31, of 12 clients 8 recovered and 4 were evicted. Mar 16 12:50:09 lola-9 kernel: Lustre: soaked-MDT0000: Recovery over after 0:06, of 12 clients 0 recovered and 12 were evicted. Mar 16 13:15:01 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 7:30, of 12 clients 7 recovered and 5 were evicted. Mar 16 14:08:40 lola-9 kernel: Lustre: soaked-MDT0000: Recovery over after 0:18, of 12 clients 0 recovered and 12 were evicted. Mar 16 15:12:03 lola-9 kernel: Lustre: soaked-MDT0000: Recovery over after 0:10, of 12 clients 0 recovered and 12 were evicted. Mar 16 18:05:10 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 0:33, of 12 clients 12 recovered and 0 were evicted. Mar 16 18:15:21 lola-9 kernel: Lustre: soaked-MDT0000: Recovery over after 0:16, of 12 clients 0 recovered and 12 were evicted. Mar 16 20:28:22 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 0:54, of 12 clients 12 recovered and 0 were evicted. Mar 16 22:14:04 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 0:47, of 12 clients 12 recovered and 0 were evicted. Mar 16 23:08:53 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 1:14, of 12 clients 12 recovered and 0 were evicted. Mar 17 00:44:31 lola-9 kernel: Lustre: soaked-MDT0000: Recovery over after 0:18, of 12 clients 0 recovered and 12 were evicted. Mar 17 02:29:42 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 1:22, of 12 clients 12 recovered and 0 were evicted. Mar 17 04:06:07 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 0:31, of 12 clients 12 recovered and 0 were evicted. Mar 17 06:24:21 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 1:05, of 12 clients 12 recovered and 0 were evicted. Mar 17 06:59:20 lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 0:53, of 12 clients 12 recovered and 0 were evicted. Mar 15 08:28:10 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 1:14, of 12 clients 12 recovered and 0 were evicted. Mar 15 09:48:55 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 0:37, of 12 clients 12 recovered and 0 were evicted. Mar 15 12:31:13 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 0:13, of 12 clients 12 recovered and 0 were evicted. Mar 15 13:29:50 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 1:09, of 12 clients 12 recovered and 0 were evicted. Mar 15 19:05:13 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 1:07, of 12 clients 12 recovered and 0 were evicted. Mar 15 20:56:41 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 0:56, of 12 clients 12 recovered and 0 were evicted. Mar 15 23:31:42 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 0:53, of 12 clients 12 recovered and 0 were evicted. Mar 16 01:12:08 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 0:56, of 12 clients 12 recovered and 0 were evicted. Mar 16 03:57:14 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 1:26, of 12 clients 12 recovered and 0 were evicted. Mar 16 06:06:11 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 0:31, of 12 clients 12 recovered and 0 were evicted. Mar 16 08:18:22 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 1:08, of 12 clients 12 recovered and 0 were evicted. Mar 16 09:05:58 lola-10 kernel: Lustre: soaked-MDT0003: Recovery over after 0:18, of 12 clients 0 recovered and 12 were evicted. Mar 16 11:59:44 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 8:45, of 13 clients 7 recovered and 6 were evicted. Mar 16 14:27:10 lola-10 kernel: Lustre: soaked-MDT0003: Recovery over after 0:20, of 12 clients 0 recovered and 12 were evicted. Mar 16 17:18:37 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 7:07, of 12 clients 12 recovered and 0 were evicted. Mar 16 19:25:17 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 0:58, of 12 clients 12 recovered and 0 were evicted. Mar 16 20:03:19 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 1:07, of 12 clients 12 recovered and 0 were evicted. Mar 16 21:08:48 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 1:06, of 12 clients 12 recovered and 0 were evicted. Mar 16 22:55:06 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 7:32, of 12 clients 2 recovered and 10 were evicted. Mar 16 23:29:44 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 0:48, of 12 clients 12 recovered and 0 were evicted. Mar 17 00:03:45 lola-10 kernel: Lustre: soaked-MDT0003: Recovery over after 0:15, of 12 clients 0 recovered and 12 were evicted. Mar 17 04:33:04 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 0:41, of 12 clients 12 recovered and 0 were evicted. Mar 17 05:15:25 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 1:21, of 12 clients 12 recovered and 0 were evicted. Mar 17 07:34:29 lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 1:16, of 12 clients 12 recovered and 0 were evicted. Mar 15 10:31:32 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 0:54, of 12 clients 12 recovered and 0 were evicted. Mar 15 12:54:28 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 0:33, of 12 clients 12 recovered and 0 were evicted. Mar 15 14:36:39 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 1:17, of 12 clients 12 recovered and 0 were evicted. Mar 15 15:30:26 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 1:07, of 12 clients 12 recovered and 0 were evicted. Mar 15 18:23:22 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 1:16, of 12 clients 12 recovered and 0 were evicted. Mar 15 19:41:12 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 0:48, of 12 clients 12 recovered and 0 were evicted. Mar 15 20:12:56 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 1:28, of 12 clients 12 recovered and 0 were evicted. Mar 15 22:41:19 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 0:29, of 12 clients 12 recovered and 0 were evicted. Mar 15 23:18:01 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 2:30, of 12 clients 12 recovered and 0 were evicted. Mar 16 00:19:03 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 1:20, of 12 clients 12 recovered and 0 were evicted. Mar 16 02:52:28 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 1:07, of 12 clients 12 recovered and 0 were evicted. Mar 16 05:04:27 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 1:00, of 12 clients 12 recovered and 0 were evicted. Mar 16 09:06:52 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 0:33, of 12 clients 12 recovered and 0 were evicted. Mar 16 11:50:45 lola-11 kernel: Lustre: soaked-MDT0002: Recovery over after 0:15, of 12 clients 0 recovered and 12 were evicted. Mar 16 14:34:52 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 7:22, of 12 clients 12 recovered and 0 were evicted. Mar 16 16:19:06 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 1:14, of 12 clients 12 recovered and 0 were evicted. Mar 16 17:11:17 lola-11 kernel: Lustre: soaked-MDT0002: Recovery over after 0:21, of 12 clients 0 recovered and 12 were evicted. Mar 16 17:58:27 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 2:55, of 12 clients 11 recovered and 1 was evicted. Mar 16 22:47:22 lola-11 kernel: Lustre: soaked-MDT0002: Recovery over after 0:16, of 12 clients 0 recovered and 12 were evicted. Mar 17 00:11:40 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 7:35, of 12 clients 4 recovered and 8 were evicted. Mar 17 03:13:31 lola-11 kernel: Lustre: soaked-MDT0003: Recovery over after 0:53, of 12 clients 12 recovered and 0 were evicted. Mar 17 04:32:11 lola-11 kernel: Lustre: soaked-MDT0002: Recovery over after 0:17, of 12 clients 0 recovered and 12 were evicted. Mar 17 07:32:49 lola-11 kernel: Lustre: soaked-MDT0002: Recovery over after 1:14, of 12 clients 12 recovered and 0 were evicted.