Aug 21 00:11:14 atlas-oss1c7.ccs.ornl.gov kernel: [3744898.536007] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 00:11:14 atlas-oss1c7.ccs.ornl.gov kernel: [3744898.562661] LustreError: Skipped 5 previous similar messages Aug 21 00:11:16 atlas-oss1c7.ccs.ornl.gov kernel: [3744900.550364] Lustre: atlas1-OST00a6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 00:11:16 atlas-oss1c7.ccs.ornl.gov kernel: [3744900.573618] Lustre: Skipped 10 previous similar messages Aug 21 00:21:18 atlas-oss1c7.ccs.ornl.gov kernel: [3745502.920883] LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 00:21:18 atlas-oss1c7.ccs.ornl.gov kernel: [3745502.951851] LustreError: Skipped 2 previous similar messages Aug 21 00:21:35 atlas-oss1c7.ccs.ornl.gov kernel: [3745519.447627] Lustre: atlas1-OST01c6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 00:21:35 atlas-oss1c7.ccs.ornl.gov kernel: [3745519.478257] Lustre: Skipped 11 previous similar messages Aug 21 00:33:17 atlas-oss1c7.ccs.ornl.gov kernel: [3746222.178952] Lustre: atlas1-OST01c6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 00:33:17 atlas-oss1c7.ccs.ornl.gov kernel: [3746222.204644] Lustre: Skipped 7 previous similar messages Aug 21 00:35:29 atlas-oss1c7.ccs.ornl.gov kernel: [3746353.826212] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 00:35:29 atlas-oss1c7.ccs.ornl.gov kernel: [3746353.854769] LustreError: Skipped 2 previous similar messages Aug 21 00:44:56 atlas-oss1c7.ccs.ornl.gov kernel: [3746921.043830] Lustre: atlas1-OST0256: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 00:44:56 atlas-oss1c7.ccs.ornl.gov kernel: [3746921.069673] Lustre: Skipped 8 previous similar messages Aug 21 00:47:26 atlas-oss1c7.ccs.ornl.gov kernel: [3747071.134458] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 00:47:26 atlas-oss1c7.ccs.ornl.gov kernel: [3747071.165813] LustreError: Skipped 6 previous similar messages Aug 21 00:59:42 atlas-oss1c7.ccs.ornl.gov kernel: [3747808.185175] Lustre: atlas1-OST0376: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 00:59:42 atlas-oss1c7.ccs.ornl.gov kernel: [3747808.216291] Lustre: Skipped 10 previous similar messages Aug 21 01:08:06 atlas-oss1c7.ccs.ornl.gov kernel: [3748311.453011] Lustre: atlas1-OST0376: haven't heard from client d5bd86e7-c566-3b25-5d41-9d23cbec2ab6 (at 1670@gni103) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8803a42fcc00, cur 1408597686 expire 1408596786 last 1408596334 Aug 21 01:08:06 atlas-oss1c7.ccs.ornl.gov kernel: [3748311.517053] Lustre: atlas1-OST0376: haven't heard from client 44983cb2-db53-b862-422d-b747e40ce861 (at 1056@gni109) in 1351 seconds. I think it's dead, and I am evicting it. exp ffff8803450fd000, cur 1408597686 expire 1408596786 last 1408596335 Aug 21 01:09:47 atlas-oss1c7.ccs.ornl.gov kernel: [3748412.472424] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 01:12:17 atlas-oss1c7.ccs.ornl.gov kernel: [3748562.528860] LustreError: 137-5: atlas1-OST0372_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 01:14:11 atlas-oss1c7.ccs.ornl.gov kernel: [3748676.575554] Lustre: atlas1-OST01c6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 01:14:11 atlas-oss1c7.ccs.ornl.gov kernel: [3748676.605562] Lustre: Skipped 6 previous similar messages Aug 21 01:21:44 atlas-oss1c7.ccs.ornl.gov kernel: [3749129.867672] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 01:24:27 atlas-oss1c7.ccs.ornl.gov kernel: [3749293.075209] Lustre: atlas1-OST02e6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 01:24:27 atlas-oss1c7.ccs.ornl.gov kernel: [3749293.099374] Lustre: Skipped 10 previous similar messages Aug 21 01:26:56 atlas-oss1c7.ccs.ornl.gov kernel: [3749442.191115] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 01:36:23 atlas-oss1c7.ccs.ornl.gov kernel: [3750009.881900] Lustre: atlas1-OST02e6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 01:36:23 atlas-oss1c7.ccs.ornl.gov kernel: [3750009.910870] Lustre: Skipped 8 previous similar messages Aug 21 01:38:53 atlas-oss1c7.ccs.ornl.gov kernel: [3750159.976260] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 01:38:53 atlas-oss1c7.ccs.ornl.gov kernel: [3750160.007798] LustreError: Skipped 7 previous similar messages Aug 21 01:48:09 atlas-oss1c7.ccs.ornl.gov kernel: [3750715.637017] Lustre: atlas1-OST0016: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 01:48:09 atlas-oss1c7.ccs.ornl.gov kernel: [3750715.668992] Lustre: Skipped 5 previous similar messages Aug 21 01:50:53 atlas-oss1c7.ccs.ornl.gov kernel: [3750880.338655] LustreError: 137-5: atlas1-OST0372_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 01:50:53 atlas-oss1c7.ccs.ornl.gov kernel: [3750880.361762] LustreError: Skipped 4 previous similar messages Aug 21 02:00:20 atlas-oss1c7.ccs.ornl.gov kernel: [3751446.671615] Lustre: atlas1-OST02e6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 02:00:20 atlas-oss1c7.ccs.ornl.gov kernel: [3751446.695939] Lustre: Skipped 13 previous similar messages Aug 21 02:19:34 atlas-oss1c7.ccs.ornl.gov kernel: [3752601.782873] LustreError: 137-5: atlas1-OST0372_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 02:19:34 atlas-oss1c7.ccs.ornl.gov kernel: [3752601.813966] LustreError: Skipped 2 previous similar messages Aug 21 02:19:35 atlas-oss1c7.ccs.ornl.gov kernel: [3752602.831654] Lustre: atlas1-OST0016: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 02:19:35 atlas-oss1c7.ccs.ornl.gov kernel: [3752602.863577] Lustre: Skipped 8 previous similar messages Aug 21 02:31:45 atlas-oss1c7.ccs.ornl.gov kernel: [3753332.519064] Lustre: atlas1-OST0376: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 02:31:45 atlas-oss1c7.ccs.ornl.gov kernel: [3753332.550866] Lustre: Skipped 10 previous similar messages Aug 21 02:34:14 atlas-oss1c7.ccs.ornl.gov kernel: [3753481.644856] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 02:34:14 atlas-oss1c7.ccs.ornl.gov kernel: [3753481.668477] LustreError: Skipped 1 previous similar message Aug 21 02:41:11 atlas-oss1c7.ccs.ornl.gov kernel: [3753898.899227] LustreError: 137-5: atlas1-OST0372_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 02:41:11 atlas-oss1c7.ccs.ornl.gov kernel: [3753898.925880] LustreError: Skipped 1 previous similar message Aug 21 02:43:41 atlas-oss1c7.ccs.ornl.gov kernel: [3754048.957603] LustreError: 137-5: atlas1-OST0132_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 02:43:41 atlas-oss1c7.ccs.ornl.gov kernel: [3754048.961563] Lustre: atlas1-OST0136: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 02:43:41 atlas-oss1c7.ccs.ornl.gov kernel: [3754048.961566] Lustre: Skipped 6 previous similar messages Aug 21 02:43:41 atlas-oss1c7.ccs.ornl.gov kernel: [3754049.033254] LustreError: Skipped 1 previous similar message Aug 21 02:46:12 atlas-oss1c7.ccs.ornl.gov kernel: [3754200.054828] LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 02:46:12 atlas-oss1c7.ccs.ornl.gov kernel: [3754200.079899] LustreError: Skipped 1 previous similar message Aug 21 02:55:40 atlas-oss1c7.ccs.ornl.gov kernel: [3754768.372037] Lustre: atlas1-OST0136: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 02:55:40 atlas-oss1c7.ccs.ornl.gov kernel: [3754768.395731] Lustre: Skipped 10 previous similar messages Aug 21 03:10:28 atlas-oss1c7.ccs.ornl.gov kernel: [3755656.676021] Lustre: atlas1-OST0256: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 03:10:28 atlas-oss1c7.ccs.ornl.gov kernel: [3755656.702421] Lustre: Skipped 4 previous similar messages Aug 21 03:22:25 atlas-oss1c7.ccs.ornl.gov kernel: [3756374.063014] Lustre: atlas1-OST0256: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 03:22:25 atlas-oss1c7.ccs.ornl.gov kernel: [3756374.094665] Lustre: Skipped 5 previous similar messages Aug 21 03:32:39 atlas-oss1c7.ccs.ornl.gov kernel: [3756987.721305] Lustre: atlas1-OST00a6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 03:32:39 atlas-oss1c7.ccs.ornl.gov kernel: [3756987.747375] Lustre: Skipped 10 previous similar messages Aug 21 03:35:07 atlas-oss1c7.ccs.ornl.gov kernel: [3757135.855627] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 03:35:07 atlas-oss1c7.ccs.ornl.gov kernel: [3757135.883202] LustreError: Skipped 1 previous similar message Aug 21 03:37:37 atlas-oss1c7.ccs.ornl.gov kernel: [3757285.891424] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 03:37:37 atlas-oss1c7.ccs.ornl.gov kernel: [3757285.920261] LustreError: Skipped 1 previous similar message Aug 21 03:42:04 atlas-oss1c7.ccs.ornl.gov kernel: [3757553.037924] LustreError: 137-5: atlas1-OST0252_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 03:44:36 atlas-oss1c7.ccs.ornl.gov kernel: [3757705.076680] Lustre: atlas1-OST00a6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 03:44:36 atlas-oss1c7.ccs.ornl.gov kernel: [3757705.099928] Lustre: Skipped 7 previous similar messages Aug 21 03:47:04 atlas-oss1c7.ccs.ornl.gov kernel: [3757853.139960] LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 03:47:04 atlas-oss1c7.ccs.ornl.gov kernel: [3757853.165351] LustreError: Skipped 1 previous similar message Aug 21 03:49:35 atlas-oss1c7.ccs.ornl.gov kernel: [3758004.210279] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 03:54:02 atlas-oss1c7.ccs.ornl.gov kernel: [3758271.365875] LustreError: 137-5: atlas1-OST0252_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 03:54:02 atlas-oss1c7.ccs.ornl.gov kernel: [3758271.393992] LustreError: Skipped 2 previous similar messages Aug 21 03:56:23 atlas-oss1c7.ccs.ornl.gov kernel: [3758413.175335] Lustre: atlas1-OST01c6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 03:56:23 atlas-oss1c7.ccs.ornl.gov kernel: [3758413.207170] Lustre: Skipped 5 previous similar messages Aug 21 03:59:02 atlas-oss1c7.ccs.ornl.gov kernel: [3758571.486497] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 04:08:29 atlas-oss1c7.ccs.ornl.gov kernel: [3759138.775155] Lustre: atlas1-OST02e6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 04:08:29 atlas-oss1c7.ccs.ornl.gov kernel: [3759138.802977] Lustre: Skipped 15 previous similar messages Aug 21 04:10:59 atlas-oss1c7.ccs.ornl.gov kernel: [3759288.858340] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 04:10:59 atlas-oss1c7.ccs.ornl.gov kernel: [3759288.889808] LustreError: Skipped 5 previous similar messages Aug 21 04:20:26 atlas-oss1c7.ccs.ornl.gov kernel: [3759856.083214] Lustre: atlas1-OST0136: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 04:20:26 atlas-oss1c7.ccs.ornl.gov kernel: [3759856.115146] Lustre: Skipped 11 previous similar messages Aug 21 04:22:58 atlas-oss1c7.ccs.ornl.gov kernel: [3760008.158410] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 04:22:58 atlas-oss1c7.ccs.ornl.gov kernel: [3760008.182579] LustreError: Skipped 5 previous similar messages Aug 21 04:32:25 atlas-oss1c7.ccs.ornl.gov kernel: [3760575.364338] Lustre: atlas1-OST00a6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 04:32:25 atlas-oss1c7.ccs.ornl.gov kernel: [3760575.387665] Lustre: Skipped 11 previous similar messages Aug 21 04:37:27 atlas-oss1c7.ccs.ornl.gov kernel: [3760877.636279] LustreError: 137-5: atlas1-OST0132_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 04:37:27 atlas-oss1c7.ccs.ornl.gov kernel: [3760877.662211] LustreError: Skipped 4 previous similar messages Aug 21 04:44:22 atlas-oss1c7.ccs.ornl.gov kernel: [3761292.793421] Lustre: atlas1-OST0256: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 04:44:22 atlas-oss1c7.ccs.ornl.gov kernel: [3761292.819883] Lustre: Skipped 8 previous similar messages Aug 21 04:58:52 atlas-oss1c7.ccs.ornl.gov kernel: [3762163.176065] Lustre: atlas1-OST0136: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 04:58:52 atlas-oss1c7.ccs.ornl.gov kernel: [3762163.199814] Lustre: Skipped 9 previous similar messages Aug 21 05:08:53 atlas-oss1c7.ccs.ornl.gov kernel: [3762764.444962] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 05:08:53 atlas-oss1c7.ccs.ornl.gov kernel: [3762764.468372] LustreError: Skipped 3 previous similar messages Aug 21 05:11:50 atlas-oss1c7.ccs.ornl.gov kernel: [3762941.826176] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 05:13:20 atlas-oss1c7.ccs.ornl.gov kernel: [3763031.605753] Lustre: atlas1-OST0136: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 05:13:20 atlas-oss1c7.ccs.ornl.gov kernel: [3763031.629359] Lustre: Skipped 13 previous similar messages Aug 21 05:23:47 atlas-oss1c7.ccs.ornl.gov kernel: [3763658.257164] Lustre: atlas1-OST02e6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 05:23:47 atlas-oss1c7.ccs.ornl.gov kernel: [3763658.286977] Lustre: Skipped 12 previous similar messages Aug 21 05:35:45 atlas-oss1c7.ccs.ornl.gov kernel: [3764376.808916] Lustre: atlas1-OST02e6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 05:35:45 atlas-oss1c7.ccs.ornl.gov kernel: [3764376.839445] Lustre: Skipped 3 previous similar messages Aug 21 05:38:16 atlas-oss1c7.ccs.ornl.gov kernel: [3764527.871602] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 05:38:16 atlas-oss1c7.ccs.ornl.gov kernel: [3764527.896354] LustreError: Skipped 1 previous similar message Aug 21 05:40:46 atlas-oss1c7.ccs.ornl.gov kernel: [3764677.857259] LustreError: 137-5: atlas1-OST0372_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 05:47:44 atlas-oss1c7.ccs.ornl.gov kernel: [3765096.220590] Lustre: atlas1-OST0376: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 05:47:44 atlas-oss1c7.ccs.ornl.gov kernel: [3765096.252116] Lustre: Skipped 14 previous similar messages Aug 21 05:47:49 atlas-oss1c7.ccs.ornl.gov kernel: [3765101.749999] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 05:47:49 atlas-oss1c7.ccs.ornl.gov kernel: [3765101.773960] LustreError: Skipped 2 previous similar messages Aug 21 05:50:13 atlas-oss1c7.ccs.ornl.gov kernel: [3765245.276608] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 05:50:13 atlas-oss1c7.ccs.ornl.gov kernel: [3765245.308079] LustreError: Skipped 1 previous similar message Aug 21 05:59:47 atlas-oss1c7.ccs.ornl.gov kernel: [3765819.075624] LustreError: 137-5: atlas1-OST0372_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 05:59:47 atlas-oss1c7.ccs.ornl.gov kernel: [3765819.106169] LustreError: Skipped 2 previous similar messages Aug 21 05:59:50 atlas-oss1c7.ccs.ornl.gov kernel: [3765823.011238] Lustre: atlas1-OST0136: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 05:59:50 atlas-oss1c7.ccs.ornl.gov kernel: [3765823.037820] Lustre: Skipped 8 previous similar messages Aug 21 06:09:13 atlas-oss1c7.ccs.ornl.gov kernel: [3766385.360655] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 06:09:13 atlas-oss1c7.ccs.ornl.gov kernel: [3766385.390731] LustreError: Skipped 1 previous similar message Aug 21 06:13:40 atlas-oss1c7.ccs.ornl.gov kernel: [3766652.433657] Lustre: atlas1-OST02e6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 06:13:40 atlas-oss1c7.ccs.ornl.gov kernel: [3766652.462434] Lustre: Skipped 6 previous similar messages Aug 21 06:24:07 atlas-oss1c7.ccs.ornl.gov kernel: [3767280.355789] Lustre: atlas1-OST0016: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 06:24:07 atlas-oss1c7.ccs.ornl.gov kernel: [3767280.380523] Lustre: Skipped 7 previous similar messages Aug 21 06:38:33 atlas-oss1c7.ccs.ornl.gov kernel: [3768145.977020] Lustre: atlas1-OST01c6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 06:38:33 atlas-oss1c7.ccs.ornl.gov kernel: [3768146.008643] Lustre: Skipped 1 previous similar message Aug 21 06:46:06 atlas-oss1c7.ccs.ornl.gov kernel: [3768599.302667] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 06:46:06 atlas-oss1c7.ccs.ornl.gov kernel: [3768599.330463] LustreError: Skipped 1 previous similar message Aug 21 06:50:30 atlas-oss1c7.ccs.ornl.gov kernel: [3768863.314925] Lustre: atlas1-OST01c6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 06:50:30 atlas-oss1c7.ccs.ornl.gov kernel: [3768863.340733] Lustre: Skipped 6 previous similar messages Aug 21 06:55:32 atlas-oss1c7.ccs.ornl.gov kernel: [3769165.633565] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 06:58:03 atlas-oss1c7.ccs.ornl.gov kernel: [3769316.707385] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 06:58:03 atlas-oss1c7.ccs.ornl.gov kernel: [3769316.732351] LustreError: Skipped 1 previous similar message Aug 21 07:02:28 atlas-oss1c7.ccs.ornl.gov kernel: [3769581.904365] Lustre: atlas1-OST00a6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 07:02:28 atlas-oss1c7.ccs.ornl.gov kernel: [3769581.933275] Lustre: Skipped 11 previous similar messages Aug 21 07:04:58 atlas-oss1c7.ccs.ornl.gov kernel: [3769731.965026] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 07:14:36 atlas-oss1c7.ccs.ornl.gov kernel: [3770310.170112] Lustre: atlas1-OST0136: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 07:14:36 atlas-oss1c7.ccs.ornl.gov kernel: [3770310.199461] Lustre: Skipped 10 previous similar messages Aug 21 07:17:06 atlas-oss1c7.ccs.ornl.gov kernel: [3770460.218587] LustreError: 137-5: atlas1-OST0252_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 07:17:06 atlas-oss1c7.ccs.ornl.gov kernel: [3770460.256138] LustreError: Skipped 8 previous similar messages Aug 21 07:29:06 atlas-oss1c7.ccs.ornl.gov kernel: [3771180.487152] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 07:29:06 atlas-oss1c7.ccs.ornl.gov kernel: [3771180.518703] LustreError: Skipped 3 previous similar messages Aug 21 07:29:06 atlas-oss1c7.ccs.ornl.gov kernel: [3771180.530627] Lustre: atlas1-OST01c6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 07:29:06 atlas-oss1c7.ccs.ornl.gov kernel: [3771180.530630] Lustre: Skipped 14 previous similar messages Aug 21 07:43:41 atlas-oss1c7.ccs.ornl.gov kernel: [3772056.275718] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 07:43:41 atlas-oss1c7.ccs.ornl.gov kernel: [3772056.301406] LustreError: Skipped 6 previous similar messages Aug 21 07:46:11 atlas-oss1c7.ccs.ornl.gov kernel: [3772206.359843] Lustre: atlas1-OST00a6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 07:46:11 atlas-oss1c7.ccs.ornl.gov kernel: [3772206.388471] Lustre: Skipped 13 previous similar messages Aug 21 07:55:38 atlas-oss1c7.ccs.ornl.gov kernel: [3772772.826266] LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 07:55:38 atlas-oss1c7.ccs.ornl.gov kernel: [3772772.853100] LustreError: Skipped 2 previous similar messages Aug 21 08:00:05 atlas-oss1c7.ccs.ornl.gov kernel: [3773039.876744] Lustre: atlas1-OST02e6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 08:00:05 atlas-oss1c7.ccs.ornl.gov kernel: [3773039.904412] Lustre: Skipped 14 previous similar messages Aug 21 08:07:38 atlas-oss1c7.ccs.ornl.gov kernel: [3773493.214732] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 08:07:38 atlas-oss1c7.ccs.ornl.gov kernel: [3773493.246296] LustreError: Skipped 11 previous similar messages Aug 21 08:10:09 atlas-oss1c7.ccs.ornl.gov kernel: [3773644.302349] Lustre: atlas1-OST0136: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 08:10:09 atlas-oss1c7.ccs.ornl.gov kernel: [3773644.333845] Lustre: Skipped 13 previous similar messages Aug 21 08:19:36 atlas-oss1c7.ccs.ornl.gov kernel: [3774211.589117] LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 08:19:36 atlas-oss1c7.ccs.ornl.gov kernel: [3774211.618765] LustreError: Skipped 6 previous similar messages Aug 21 08:22:07 atlas-oss1c7.ccs.ornl.gov kernel: [3774362.636077] Lustre: atlas1-OST0016: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 08:22:07 atlas-oss1c7.ccs.ornl.gov kernel: [3774362.665982] Lustre: Skipped 16 previous similar messages Aug 21 08:31:33 atlas-oss1c7.ccs.ornl.gov kernel: [3774929.050977] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 08:31:33 atlas-oss1c7.ccs.ornl.gov kernel: [3774929.080965] LustreError: Skipped 7 previous similar messages Aug 21 08:34:07 atlas-oss1c7.ccs.ornl.gov kernel: [3775082.719425] Lustre: atlas1-OST0016: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 08:34:07 atlas-oss1c7.ccs.ornl.gov kernel: [3775082.749075] Lustre: Skipped 8 previous similar messages Aug 21 08:43:30 atlas-oss1c7.ccs.ornl.gov kernel: [3775646.456728] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 08:43:30 atlas-oss1c7.ccs.ornl.gov kernel: [3775646.482916] LustreError: Skipped 4 previous similar messages Aug 21 08:47:56 atlas-oss1c7.ccs.ornl.gov kernel: [3775912.478698] Lustre: atlas1-OST0016: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 08:47:56 atlas-oss1c7.ccs.ornl.gov kernel: [3775912.503897] Lustre: Skipped 16 previous similar messages Aug 21 08:57:59 atlas-oss1c7.ccs.ornl.gov kernel: [3776515.968296] LustreError: 137-5: atlas1-OST0252_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 08:57:59 atlas-oss1c7.ccs.ornl.gov kernel: [3776515.992555] LustreError: Skipped 8 previous similar messages Aug 21 08:58:02 atlas-oss1c7.ccs.ornl.gov kernel: [3776518.750709] Lustre: atlas1-OST0016: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 08:58:02 atlas-oss1c7.ccs.ornl.gov kernel: [3776518.774395] Lustre: Skipped 11 previous similar messages Aug 21 09:10:00 atlas-oss1c7.ccs.ornl.gov kernel: [3777237.227312] Lustre: atlas1-OST0016: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 09:10:00 atlas-oss1c7.ccs.ornl.gov kernel: [3777237.256091] Lustre: Skipped 12 previous similar messages Aug 21 09:11:52 atlas-oss1c7.ccs.ornl.gov kernel: [3777348.430467] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 09:11:52 atlas-oss1c7.ccs.ornl.gov kernel: [3777348.458081] LustreError: Skipped 5 previous similar messages Aug 21 09:21:57 atlas-oss1c7.ccs.ornl.gov kernel: [3777953.688824] LustreError: 137-5: atlas1-OST0132_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 09:21:57 atlas-oss1c7.ccs.ornl.gov kernel: [3777953.717728] LustreError: Skipped 6 previous similar messages Aug 21 09:24:27 atlas-oss1c7.ccs.ornl.gov kernel: [3778103.708380] Lustre: atlas1-OST0136: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 09:24:27 atlas-oss1c7.ccs.ornl.gov kernel: [3778103.734660] Lustre: Skipped 14 previous similar messages Aug 21 09:33:54 atlas-oss1c7.ccs.ornl.gov kernel: [3778671.042938] LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 09:33:54 atlas-oss1c7.ccs.ornl.gov kernel: [3778671.069563] LustreError: Skipped 8 previous similar messages Aug 21 09:35:45 atlas-oss1c7.ccs.ornl.gov kernel: [3778782.094729] Lustre: atlas1-OST0016: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 09:35:45 atlas-oss1c7.ccs.ornl.gov kernel: [3778782.122036] Lustre: Skipped 13 previous similar messages Aug 21 09:47:46 atlas-oss1c7.ccs.ornl.gov kernel: [3779503.517767] LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 09:47:46 atlas-oss1c7.ccs.ornl.gov kernel: [3779503.520385] Lustre: atlas1-OST02e6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 09:47:46 atlas-oss1c7.ccs.ornl.gov kernel: [3779503.520387] Lustre: Skipped 11 previous similar messages Aug 21 09:47:46 atlas-oss1c7.ccs.ornl.gov kernel: [3779503.595990] LustreError: Skipped 5 previous similar messages Aug 21 09:57:57 atlas-oss1c7.ccs.ornl.gov kernel: [3780114.567946] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 09:57:57 atlas-oss1c7.ccs.ornl.gov kernel: [3780114.597183] LustreError: Skipped 7 previous similar messages Aug 21 09:59:44 atlas-oss1c7.ccs.ornl.gov kernel: [3780222.002348] Lustre: atlas1-OST02e6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 09:59:44 atlas-oss1c7.ccs.ornl.gov kernel: [3780222.027915] Lustre: Skipped 13 previous similar messages Aug 21 10:09:54 atlas-oss1c7.ccs.ornl.gov kernel: [3780831.952791] Lustre: atlas1-OST0136: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 10:09:54 atlas-oss1c7.ccs.ornl.gov kernel: [3780831.952863] LustreError: 137-5: atlas1-OST0132_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 10:09:54 atlas-oss1c7.ccs.ornl.gov kernel: [3780831.952865] LustreError: Skipped 4 previous similar messages Aug 21 10:09:54 atlas-oss1c7.ccs.ornl.gov kernel: [3780832.029727] Lustre: Skipped 12 previous similar messages Aug 21 10:21:08 atlas-oss1c7.ccs.ornl.gov kernel: [3781506.004410] Lustre: atlas1-OST0016: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 10:21:08 atlas-oss1c7.ccs.ornl.gov kernel: [3781506.034831] Lustre: Skipped 10 previous similar messages Aug 21 10:21:54 atlas-oss1c7.ccs.ornl.gov kernel: [3781552.333126] LustreError: 137-5: atlas1-OST0372_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 10:21:54 atlas-oss1c7.ccs.ornl.gov kernel: [3781552.362234] LustreError: Skipped 2 previous similar messages Aug 21 10:31:09 atlas-oss1c7.ccs.ornl.gov kernel: [3782107.441931] Lustre: atlas1-OST02e6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 10:31:09 atlas-oss1c7.ccs.ornl.gov kernel: [3782107.472878] Lustre: Skipped 6 previous similar messages Aug 21 10:38:05 atlas-oss1c7.ccs.ornl.gov kernel: [3782524.048634] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 10:38:05 atlas-oss1c7.ccs.ornl.gov kernel: [3782524.080467] LustreError: Skipped 4 previous similar messages Aug 21 10:40:37 atlas-oss1c7.ccs.ornl.gov kernel: [3782675.402057] Lustre: atlas1-OST01c6: haven't heard from client ed0b84e7-8f16-b2e9-1ed8-7c9ddd46f415 (at 2248@gni112) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8804f7293800, cur 1408632037 expire 1408631137 last 1408630685 Aug 21 10:40:37 atlas-oss1c7.ccs.ornl.gov kernel: [3782675.467525] Lustre: Skipped 19 previous similar messages Aug 21 10:40:37 atlas-oss1c7.ccs.ornl.gov kernel: [3782675.479003] Lustre: atlas1-OST0376: haven't heard from client ed0b84e7-8f16-b2e9-1ed8-7c9ddd46f415 (at 2248@gni112) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880603c0c400, cur 1408632037 expire 1408631137 last 1408630685 Aug 21 10:43:08 atlas-oss1c7.ccs.ornl.gov kernel: [3782827.297943] Lustre: atlas1-OST01c6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 10:43:08 atlas-oss1c7.ccs.ornl.gov kernel: [3782827.326011] Lustre: Skipped 13 previous similar messages Aug 21 10:50:03 atlas-oss1c7.ccs.ornl.gov kernel: [3783242.505748] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 10:50:03 atlas-oss1c7.ccs.ornl.gov kernel: [3783242.533491] LustreError: Skipped 6 previous similar messages Aug 21 10:59:40 atlas-oss1c7.ccs.ornl.gov kernel: [3783819.462338] Lustre: atlas1-OST00a6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 10:59:40 atlas-oss1c7.ccs.ornl.gov kernel: [3783819.492103] Lustre: Skipped 15 previous similar messages Aug 21 11:02:11 atlas-oss1c7.ccs.ornl.gov kernel: [3783970.561354] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 11:02:11 atlas-oss1c7.ccs.ornl.gov kernel: [3783970.589427] LustreError: Skipped 5 previous similar messages Aug 21 11:11:38 atlas-oss1c7.ccs.ornl.gov kernel: [3784537.923284] Lustre: atlas1-OST0136: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 11:11:38 atlas-oss1c7.ccs.ornl.gov kernel: [3784537.954710] Lustre: Skipped 12 previous similar messages Aug 21 11:14:09 atlas-oss1c7.ccs.ornl.gov kernel: [3784689.042095] LustreError: 137-5: atlas1-OST02e2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 11:14:09 atlas-oss1c7.ccs.ornl.gov kernel: [3784689.071884] LustreError: Skipped 2 previous similar messages Aug 21 11:20:50 atlas-oss1c7.ccs.ornl.gov kernel: [3785089.312142] Lustre: atlas1-OST02e6: haven't heard from client 9b2f1482-1f23-fec3-c37d-67d295f143fc (at 10.36.202.175@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88092666dc00, cur 1408634450 expire 1408633550 last 1408633098 Aug 21 11:20:50 atlas-oss1c7.ccs.ornl.gov kernel: [3785089.383630] Lustre: Skipped 5 previous similar messages Aug 21 11:20:50 atlas-oss1c7.ccs.ornl.gov kernel: [3785089.403145] Lustre: atlas1-OST0016: haven't heard from client 9b2f1482-1f23-fec3-c37d-67d295f143fc (at 10.36.202.175@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880bc18c7400, cur 1408634450 expire 1408633550 last 1408633098 Aug 21 11:23:36 atlas-oss1c7.ccs.ornl.gov kernel: [3785255.412906] Lustre: atlas1-OST00a6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 11:23:36 atlas-oss1c7.ccs.ornl.gov kernel: [3785255.436612] Lustre: Skipped 11 previous similar messages Aug 21 11:26:08 atlas-oss1c7.ccs.ornl.gov kernel: [3785407.505321] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 11:26:08 atlas-oss1c7.ccs.ornl.gov kernel: [3785407.534114] LustreError: Skipped 4 previous similar messages Aug 21 11:35:33 atlas-oss1c7.ccs.ornl.gov kernel: [3785972.675882] Lustre: atlas1-OST00a6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 11:35:33 atlas-oss1c7.ccs.ornl.gov kernel: [3785972.707894] Lustre: Skipped 6 previous similar messages Aug 21 11:38:04 atlas-oss1c7.ccs.ornl.gov kernel: [3786123.752474] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 11:38:04 atlas-oss1c7.ccs.ornl.gov kernel: [3786123.775972] LustreError: Skipped 3 previous similar messages Aug 21 11:47:31 atlas-oss1c7.ccs.ornl.gov kernel: [3786691.091720] Lustre: atlas1-OST01c6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 11:47:31 atlas-oss1c7.ccs.ornl.gov kernel: [3786691.121222] Lustre: Skipped 12 previous similar messages Aug 21 11:52:40 atlas-oss1c7.ccs.ornl.gov kernel: [3787000.381594] LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 11:52:40 atlas-oss1c7.ccs.ornl.gov kernel: [3787000.408140] LustreError: Skipped 8 previous similar messages Aug 21 11:54:56 atlas-oss1c7.ccs.ornl.gov kernel: [3787136.130398] Lustre: atlas1-OST0376: haven't heard from client 29d2a0c2-fc18-9262-8e96-d75770a4df9c (at 10.36.202.175@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8801cbb16800, cur 1408636496 expire 1408635596 last 1408635144 Aug 21 11:54:56 atlas-oss1c7.ccs.ornl.gov kernel: [3787136.199672] Lustre: Skipped 5 previous similar messages Aug 21 11:57:26 atlas-oss1c7.ccs.ornl.gov kernel: [3787286.260873] Lustre: atlas1-OST0016: haven't heard from client 29d2a0c2-fc18-9262-8e96-d75770a4df9c (at 10.36.202.175@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88006bc5b800, cur 1408636646 expire 1408635746 last 1408635294 Aug 21 11:59:37 atlas-oss1c7.ccs.ornl.gov kernel: [3787417.583050] Lustre: atlas1-OST0376: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 11:59:37 atlas-oss1c7.ccs.ornl.gov kernel: [3787417.606689] Lustre: Skipped 10 previous similar messages Aug 21 12:04:39 atlas-oss1c7.ccs.ornl.gov kernel: [3787719.711155] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 12:04:39 atlas-oss1c7.ccs.ornl.gov kernel: [3787719.741127] LustreError: Skipped 4 previous similar messages Aug 21 12:11:34 atlas-oss1c7.ccs.ornl.gov kernel: [3788134.880649] Lustre: atlas1-OST02e6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 12:11:34 atlas-oss1c7.ccs.ornl.gov kernel: [3788134.908923] Lustre: Skipped 9 previous similar messages Aug 21 12:16:37 atlas-oss1c7.ccs.ornl.gov kernel: [3788438.112167] LustreError: 137-5: atlas1-OST0132_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 12:16:37 atlas-oss1c7.ccs.ornl.gov kernel: [3788438.143564] LustreError: Skipped 4 previous similar messages Aug 21 12:23:33 atlas-oss1c7.ccs.ornl.gov kernel: [3788854.215509] Lustre: atlas1-OST01c6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 12:23:33 atlas-oss1c7.ccs.ornl.gov kernel: [3788854.241170] Lustre: Skipped 13 previous similar messages Aug 21 12:28:08 atlas-oss1c7.ccs.ornl.gov kernel: [3789129.303168] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 12:28:08 atlas-oss1c7.ccs.ornl.gov kernel: [3789129.326583] LustreError: Skipped 7 previous similar messages Aug 21 12:35:43 atlas-oss1c7.ccs.ornl.gov kernel: [3789584.565018] Lustre: atlas1-OST00a6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 12:35:43 atlas-oss1c7.ccs.ornl.gov kernel: [3789584.598959] Lustre: Skipped 9 previous similar messages Aug 21 12:40:06 atlas-oss1c7.ccs.ornl.gov kernel: [3789847.731188] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 12:40:06 atlas-oss1c7.ccs.ornl.gov kernel: [3789847.758115] LustreError: Skipped 5 previous similar messages Aug 21 12:47:41 atlas-oss1c7.ccs.ornl.gov kernel: [3790302.958731] Lustre: atlas1-OST0016: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 12:47:41 atlas-oss1c7.ccs.ornl.gov kernel: [3790302.990532] Lustre: Skipped 13 previous similar messages Aug 21 12:59:59 atlas-oss1c7.ccs.ornl.gov kernel: [3791041.308437] Lustre: atlas1-OST02e6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 12:59:59 atlas-oss1c7.ccs.ornl.gov kernel: [3791041.340121] Lustre: Skipped 8 previous similar messages Aug 21 13:11:59 atlas-oss1c7.ccs.ornl.gov kernel: [3791761.699053] Lustre: atlas1-OST0256: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 13:11:59 atlas-oss1c7.ccs.ornl.gov kernel: [3791761.723905] Lustre: Skipped 5 previous similar messages Aug 21 13:19:00 atlas-oss1c7.ccs.ornl.gov kernel: [3792182.782152] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 13:19:00 atlas-oss1c7.ccs.ornl.gov kernel: [3792182.813560] LustreError: Skipped 5 previous similar messages Aug 21 13:21:30 atlas-oss1c7.ccs.ornl.gov kernel: [3792332.853481] LustreError: 137-5: atlas1-OST0252_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 13:23:56 atlas-oss1c7.ccs.ornl.gov kernel: [3792478.075820] Lustre: atlas1-OST0016: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 13:23:56 atlas-oss1c7.ccs.ornl.gov kernel: [3792478.105687] Lustre: Skipped 8 previous similar messages Aug 21 13:30:57 atlas-oss1c7.ccs.ornl.gov kernel: [3792900.091115] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 13:30:57 atlas-oss1c7.ccs.ornl.gov kernel: [3792900.115418] LustreError: Skipped 2 previous similar messages Aug 21 13:36:00 atlas-oss1c7.ccs.ornl.gov kernel: [3793203.296699] Lustre: atlas1-OST0016: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 13:36:00 atlas-oss1c7.ccs.ornl.gov kernel: [3793203.320521] Lustre: Skipped 10 previous similar messages Aug 21 13:42:56 atlas-oss1c7.ccs.ornl.gov kernel: [3793618.536274] LustreError: 137-5: atlas1-OST0012_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 13:42:56 atlas-oss1c7.ccs.ornl.gov kernel: [3793618.567851] LustreError: Skipped 4 previous similar messages Aug 21 13:47:59 atlas-oss1c7.ccs.ornl.gov kernel: [3793921.739350] Lustre: atlas1-OST01c6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 13:47:59 atlas-oss1c7.ccs.ornl.gov kernel: [3793921.763150] Lustre: Skipped 13 previous similar messages Aug 21 13:55:57 atlas-oss1c7.ccs.ornl.gov kernel: [3794399.851822] Lustre: atlas1-OST0016: haven't heard from client dc907747-c2f8-2452-bba2-1f9365c7ced9 (at 10.36.207.215@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880ba9d61000, cur 1408643757 expire 1408642857 last 1408642405 Aug 21 13:55:57 atlas-oss1c7.ccs.ornl.gov kernel: [3794399.924078] Lustre: Skipped 5 previous similar messages Aug 21 13:55:57 atlas-oss1c7.ccs.ornl.gov kernel: [3794399.943785] Lustre: atlas1-OST01c6: haven't heard from client dc907747-c2f8-2452-bba2-1f9365c7ced9 (at 10.36.207.215@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88075fcc6c00, cur 1408643757 expire 1408642857 last 1408642405 Aug 21 13:55:58 atlas-oss1c7.ccs.ornl.gov kernel: [3794400.919238] Lustre: atlas1-OST0376: haven't heard from client dc907747-c2f8-2452-bba2-1f9365c7ced9 (at 10.36.207.215@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880257041400, cur 1408643758 expire 1408642858 last 1408642405 Aug 21 13:55:58 atlas-oss1c7.ccs.ornl.gov kernel: [3794400.984690] Lustre: Skipped 3 previous similar messages Aug 21 13:55:59 atlas-oss1c7.ccs.ornl.gov kernel: [3794401.991204] Lustre: atlas1-OST02e6: haven't heard from client dc907747-c2f8-2452-bba2-1f9365c7ced9 (at 10.36.207.215@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8808cbcaa800, cur 1408643759 expire 1408642859 last 1408642405 Aug 21 14:00:23 atlas-oss1c7.ccs.ornl.gov kernel: [3794666.155556] Lustre: atlas1-OST0016: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 14:00:23 atlas-oss1c7.ccs.ornl.gov kernel: [3794666.185134] Lustre: Skipped 6 previous similar messages Aug 21 14:02:53 atlas-oss1c7.ccs.ornl.gov kernel: [3794816.201589] LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 14:12:20 atlas-oss1c7.ccs.ornl.gov kernel: [3795383.544447] Lustre: atlas1-OST0016: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 14:12:20 atlas-oss1c7.ccs.ornl.gov kernel: [3795383.567821] Lustre: Skipped 5 previous similar messages Aug 21 14:19:20 atlas-oss1c7.ccs.ornl.gov kernel: [3795803.534475] LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 14:19:20 atlas-oss1c7.ccs.ornl.gov kernel: [3795803.566134] LustreError: Skipped 3 previous similar messages Aug 21 14:26:17 atlas-oss1c7.ccs.ornl.gov kernel: [3796220.710761] Lustre: atlas1-OST01c6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 14:26:17 atlas-oss1c7.ccs.ornl.gov kernel: [3796220.734900] Lustre: Skipped 11 previous similar messages Aug 21 14:31:17 atlas-oss1c7.ccs.ornl.gov kernel: [3796520.850175] LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 14:31:17 atlas-oss1c7.ccs.ornl.gov kernel: [3796520.878544] LustreError: Skipped 4 previous similar messages Aug 21 14:38:14 atlas-oss1c7.ccs.ornl.gov kernel: [3796937.996148] Lustre: atlas1-OST0136: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting Aug 21 14:38:14 atlas-oss1c7.ccs.ornl.gov kernel: [3796938.026732] Lustre: Skipped 6 previous similar messages Aug 21 14:44:39 atlas-oss1c7.ccs.ornl.gov kernel: [3797323.165332] LustreError: 33287:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 603, more than at_max 600 Aug 21 14:44:39 atlas-oss1c7.ccs.ornl.gov kernel: [3797323.165334] ns: filter-atlas1-OST0376_UUID lock: ffff88034b20a900/0xecd0a12120f17d4c lrc: 4/0,0 mode: PW/PW res: [0x5c2a0f:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) flags: 0x10020 nid: 10.36.205.208@o2ib remote: 0xc39a22a84c361c61 expref: 26 pid: 14426 timeout: 8090929608 lvb_type: 0 Aug 21 14:44:39 atlas-oss1c7.ccs.ornl.gov kernel: [3797323.292254] LustreError: 33287:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) Skipped 3 previous similar messages Aug 21 14:44:57 atlas-oss1c7.ccs.ornl.gov kernel: [3797340.909561] Lustre: atlas1-OST0136: Slow creates, 128/256 objects created at a rate of 2/s Aug 21 14:45:53 atlas-oss1c7.ccs.ornl.gov kernel: [3797397.137591] LustreError: 137-5: atlas1-OST0372_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 14:45:53 atlas-oss1c7.ccs.ornl.gov kernel: [3797397.161035] LustreError: Skipped 2 previous similar messages Aug 21 14:48:24 atlas-oss1c7.ccs.ornl.gov kernel: [3797548.207840] Lustre: atlas1-OST01c6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 14:48:24 atlas-oss1c7.ccs.ornl.gov kernel: [3797548.238152] Lustre: Skipped 6 previous similar messages Aug 21 14:52:26 atlas-oss1c7.ccs.ornl.gov kernel: [3797790.627280] LustreError: 45633:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 610s Aug 21 14:52:26 atlas-oss1c7.ccs.ornl.gov kernel: [3797790.659551] LustreError: 45633:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 610s Aug 21 14:53:26 atlas-oss1c7.ccs.ornl.gov kernel: [3797851.076823] LustreError: 46496:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 631s Aug 21 14:54:26 atlas-oss1c7.ccs.ornl.gov kernel: [3797910.850073] LustreError: 46636:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 629s Aug 21 14:54:26 atlas-oss1c7.ccs.ornl.gov kernel: [3797910.876295] LustreError: 46636:0:(service.c:3216:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message Aug 21 14:54:27 atlas-oss1c7.ccs.ornl.gov kernel: [3797911.436996] Lustre: 33219:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/3), not sending early reply Aug 21 14:54:27 atlas-oss1c7.ccs.ornl.gov kernel: [3797911.436998] req@ffff8802f640e400 x1476723491621465/t0(0) o3->6ae34883-e4de-08ed-e3e9-ccdd41e9934d@9719@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647272 ref 2 fl Interpret:/0/0 rc 0/0 Aug 21 14:54:27 atlas-oss1c7.ccs.ornl.gov kernel: [3797911.525865] Lustre: 33219:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Aug 21 14:54:30 atlas-oss1c7.ccs.ornl.gov kernel: [3797914.439134] Lustre: 33155:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/4), not sending early reply Aug 21 14:54:30 atlas-oss1c7.ccs.ornl.gov kernel: [3797914.439136] req@ffff8801113cf800 x1476723486468127/t0(0) o3->899dc82e-351a-4bec-731c-f1dfa2c1f8bf@9697@gni108:0/0 lens 448/0 e 0 to 0 dl 1408647275 ref 2 fl New:/0/ffffffff rc 0/-1 Aug 21 14:54:30 atlas-oss1c7.ccs.ornl.gov kernel: [3797914.527175] Lustre: 33155:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 26 previous similar messages Aug 21 14:54:32 atlas-oss1c7.ccs.ornl.gov kernel: [3797916.390863] LustreError: 33334:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 0+0s req@ffff88023d27b000 x1476723483171410/t0(0) o3->711e8a57-ae3b-7204-47a9-6a996887d00c@7253@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647272 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:54:32 atlas-oss1c7.ccs.ornl.gov kernel: [3797916.390933] Lustre: atlas1-OST0136: Bulk IO read error with 711e8a57-ae3b-7204-47a9-6a996887d00c (at 7253@gni108), client will retry: rc -110 Aug 21 14:54:32 atlas-oss1c7.ccs.ornl.gov kernel: [3797916.392844] LustreError: 33261:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 0+0s req@ffff8803bab27000 x1476723483171409/t0(0) o3->711e8a57-ae3b-7204-47a9-6a996887d00c@7253@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647272 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:54:32 atlas-oss1c7.ccs.ornl.gov kernel: [3797916.392848] LustreError: 33261:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 1 previous similar message Aug 21 14:54:32 atlas-oss1c7.ccs.ornl.gov kernel: [3797916.392913] Lustre: atlas1-OST0136: Bulk IO read error with 711e8a57-ae3b-7204-47a9-6a996887d00c (at 7253@gni108), client will retry: rc -110 Aug 21 14:54:34 atlas-oss1c7.ccs.ornl.gov kernel: [3797918.547680] Lustre: 33239:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/2), not sending early reply Aug 21 14:54:34 atlas-oss1c7.ccs.ornl.gov kernel: [3797918.547682] req@ffff880c0290e000 x1476723485462921/t0(0) o3->85d016ef-95df-ec09-bd4c-35466fd2d83e@2033@gni109:0/0 lens 448/432 e 0 to 0 dl 1408647279 ref 2 fl Interpret:/0/0 rc 0/0 Aug 21 14:54:34 atlas-oss1c7.ccs.ornl.gov kernel: [3797918.638452] Lustre: 33239:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 337 previous similar messages Aug 21 14:54:36 atlas-oss1c7.ccs.ornl.gov kernel: [3797920.709503] LustreError: 33297:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -4+4s req@ffff8803f8938000 x1476723491621464/t0(0) o3->6ae34883-e4de-08ed-e3e9-ccdd41e9934d@9719@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647272 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:54:36 atlas-oss1c7.ccs.ornl.gov kernel: [3797920.717576] Lustre: atlas1-OST0136: Bulk IO read error with 995293b6-49ae-5653-9559-baf2afd203ba (at 1093@gni109), client will retry: rc -110 Aug 21 14:54:36 atlas-oss1c7.ccs.ornl.gov kernel: [3797920.717579] Lustre: Skipped 2 previous similar messages Aug 21 14:54:36 atlas-oss1c7.ccs.ornl.gov kernel: [3797920.717589] Lustre: 33065:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (602:4s); client may timeout. req@ffff880c8f484000 x1476723496245724/t0(0) o3->995293b6-49ae-5653-9559-baf2afd203ba@1093@gni109:0/0 lens 448/432 e 0 to 0 dl 1408647272 ref 1 fl Complete:/0/ffffffff rc 0/-1 Aug 21 14:54:36 atlas-oss1c7.ccs.ornl.gov kernel: [3797920.940119] LustreError: 33297:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 5 previous similar messages Aug 21 14:54:41 atlas-oss1c7.ccs.ornl.gov kernel: [3797925.037118] LustreError: 33191:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -1+1s req@ffff880aeee84c00 x1476723495010102/t0(0) o3->86f683b4-fc92-a3cd-b12e-1b8013211fe8@2517@gni112:0/0 lens 448/432 e 0 to 0 dl 1408647279 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:54:41 atlas-oss1c7.ccs.ornl.gov kernel: [3797925.043401] Lustre: atlas1-OST0256: Bulk IO read error with 4b8311aa-a160-bdc3-d939-baad15e85465 (at 9652@gni108), client will retry: rc -110 Aug 21 14:54:41 atlas-oss1c7.ccs.ornl.gov kernel: [3797925.043404] Lustre: Skipped 4 previous similar messages Aug 21 14:54:41 atlas-oss1c7.ccs.ornl.gov kernel: [3797925.043413] Lustre: 33045:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (603:2s); client may timeout. req@ffff88004b2ab000 x1476723487358404/t0(0) o3->4b8311aa-a160-bdc3-d939-baad15e85465@9652@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647278 ref 1 fl Complete:/0/ffffffff rc 0/-1 Aug 21 14:54:41 atlas-oss1c7.ccs.ornl.gov kernel: [3797925.043417] Lustre: 33045:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 5 previous similar messages Aug 21 14:54:41 atlas-oss1c7.ccs.ornl.gov kernel: [3797925.101693] LustreError: 33254:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-7194@gni108: deadline 601:5s ago Aug 21 14:54:41 atlas-oss1c7.ccs.ornl.gov kernel: [3797925.101695] req@ffff8804fadfec00 x1476723485267824/t0(0) o3->9ac3cc28-5d33-a98f-9d7e-cd610cf158f4@7194@gni108:0/0 lens 448/0 e 0 to 0 dl 1408647275 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Aug 21 14:54:41 atlas-oss1c7.ccs.ornl.gov kernel: [3797925.402356] LustreError: 33191:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 10 previous similar messages Aug 21 14:54:41 atlas-oss1c7.ccs.ornl.gov kernel: [3797925.403269] LustreError: 33259:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-8643@gni111: deadline 605:1s ago Aug 21 14:54:41 atlas-oss1c7.ccs.ornl.gov kernel: [3797925.403271] req@ffff880b7d778c00 x1476723496108215/t0(0) o3->03aa7cb2-f5a5-93ff-9162-e991e7e2d8a7@8643@gni111:0/0 lens 448/0 e 0 to 0 dl 1408647280 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Aug 21 14:54:41 atlas-oss1c7.ccs.ornl.gov kernel: [3797925.403274] LustreError: 33259:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 283 previous similar messages Aug 21 14:54:43 atlas-oss1c7.ccs.ornl.gov kernel: [3797927.443055] Lustre: 33219:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/0), not sending early reply Aug 21 14:54:43 atlas-oss1c7.ccs.ornl.gov kernel: [3797927.443057] req@ffff880466e0e400 x1476723490530284/t0(0) o3->12345d68-e7b8-c146-d144-44cd20a60b66@7240@gni108:0/0 lens 448/0 e 0 to 0 dl 1408647288 ref 2 fl New:/0/ffffffff rc 0/-1 Aug 21 14:54:43 atlas-oss1c7.ccs.ornl.gov kernel: [3797927.541776] Lustre: 33219:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 275 previous similar messages Aug 21 14:54:45 atlas-oss1c7.ccs.ornl.gov kernel: [3797929.374753] LustreError: 33056:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -7+7s req@ffff880b59107800 x1476723489864103/t0(0) o3->96af42e6-9f7d-d26a-8f2b-9bb0cbd2ff06@2502@gni112:0/0 lens 448/432 e 0 to 0 dl 1408647278 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:54:45 atlas-oss1c7.ccs.ornl.gov kernel: [3797929.452869] Lustre: atlas1-OST0256: Bulk IO read error with 6ae34883-e4de-08ed-e3e9-ccdd41e9934d (at 9719@gni108), client will retry: rc -110 Aug 21 14:54:45 atlas-oss1c7.ccs.ornl.gov kernel: [3797929.452918] Lustre: 33205:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (603:3s); client may timeout. req@ffff880d95e37000 x1476723492722369/t0(0) o3->4bc39016-910b-7999-0155-877edfaae735@9767@gni111:0/0 lens 448/432 e 0 to 0 dl 1408647282 ref 1 fl Complete:/0/ffffffff rc 0/-1 Aug 21 14:54:45 atlas-oss1c7.ccs.ornl.gov kernel: [3797929.452922] Lustre: 33205:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 443 previous similar messages Aug 21 14:54:45 atlas-oss1c7.ccs.ornl.gov kernel: [3797929.453194] LustreError: 33056:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 2 previous similar messages Aug 21 14:54:45 atlas-oss1c7.ccs.ornl.gov kernel: [3797929.454159] LustreError: 33296:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-7122@gni111: deadline 605:4s ago Aug 21 14:54:45 atlas-oss1c7.ccs.ornl.gov kernel: [3797929.454161] req@ffff880e46320800 x1476723501101999/t0(0) o3->39c9de6f-c220-aa62-9f39-6cbf860ccc16@7122@gni111:0/0 lens 448/0 e 0 to 0 dl 1408647281 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Aug 21 14:54:45 atlas-oss1c7.ccs.ornl.gov kernel: [3797929.454164] LustreError: 33296:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 146 previous similar messages Aug 21 14:54:45 atlas-oss1c7.ccs.ornl.gov kernel: [3797929.793063] Lustre: Skipped 52 previous similar messages Aug 21 14:54:49 atlas-oss1c7.ccs.ornl.gov kernel: [3797933.705432] LustreError: 33122:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -9+9s req@ffff8803bc4c8400 x1476723478179511/t0(0) o3->33a957a7-9af1-fbe4-73c6-f08cc6eed1ab@8164@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647280 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:54:49 atlas-oss1c7.ccs.ornl.gov kernel: [3797933.706456] Lustre: atlas1-OST0136: Bulk IO read error with 33a957a7-9af1-fbe4-73c6-f08cc6eed1ab (at 8164@gni108), client will retry: rc -110 Aug 21 14:54:49 atlas-oss1c7.ccs.ornl.gov kernel: [3797933.706459] Lustre: Skipped 3 previous similar messages Aug 21 14:54:49 atlas-oss1c7.ccs.ornl.gov kernel: [3797933.706469] Lustre: 33153:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (605:9s); client may timeout. req@ffff880810755800 x1476723478179510/t0(0) o3->33a957a7-9af1-fbe4-73c6-f08cc6eed1ab@8164@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647280 ref 1 fl Complete:/0/ffffffff rc 0/-1 Aug 21 14:54:49 atlas-oss1c7.ccs.ornl.gov kernel: [3797933.706472] Lustre: 33153:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 85 previous similar messages Aug 21 14:54:49 atlas-oss1c7.ccs.ornl.gov kernel: [3797933.975617] LustreError: 33122:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 78 previous similar messages Aug 21 14:54:58 atlas-oss1c7.ccs.ornl.gov kernel: [3797942.366627] LustreError: 33358:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 0+0s req@ffff8804ce58b400 x1476723486311220/t0(0) o3->41af8c90-384f-f0ae-b78d-98ca74f9a26e@9702@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647298 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:54:58 atlas-oss1c7.ccs.ornl.gov kernel: [3797942.446858] LustreError: 33358:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 39 previous similar messages Aug 21 14:54:58 atlas-oss1c7.ccs.ornl.gov kernel: [3797942.477381] Lustre: atlas1-OST0256: Bulk IO read error with 41af8c90-384f-f0ae-b78d-98ca74f9a26e (at 9702@gni108), client will retry: rc -110 Aug 21 14:54:58 atlas-oss1c7.ccs.ornl.gov kernel: [3797942.517626] Lustre: Skipped 79 previous similar messages Aug 21 14:55:02 atlas-oss1c7.ccs.ornl.gov kernel: [3797946.451236] Lustre: 33137:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/0), not sending early reply Aug 21 14:55:02 atlas-oss1c7.ccs.ornl.gov kernel: [3797946.451238] req@ffff8808a8fa5400 x1476723490499440/t0(0) o4->ebdd8811-b12e-29e0-3368-e494bac18b79@11514@gni102:0/0 lens 448/448 e 0 to 0 dl 1408647307 ref 2 fl Interpret:/0/0 rc 0/0 Aug 21 14:55:02 atlas-oss1c7.ccs.ornl.gov kernel: [3797946.539351] Lustre: 33137:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 201 previous similar messages Aug 21 14:55:06 atlas-oss1c7.ccs.ornl.gov kernel: [3797951.025297] Lustre: 33139:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (605:5s); client may timeout. req@ffff880e5b22cc00 x1476723490981567/t0(0) o3->62068cc7-602f-e25c-05f6-da3dcdefb5a5@1039@gni109:0/0 lens 448/432 e 0 to 0 dl 1408647301 ref 1 fl Complete:/0/ffffffff rc 0/-1 Aug 21 14:55:06 atlas-oss1c7.ccs.ornl.gov kernel: [3797951.111085] Lustre: 33139:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 81 previous similar messages Aug 21 14:56:28 atlas-oss1c7.ccs.ornl.gov kernel: [3798032.234129] LustreError: 47170:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 617s Aug 21 14:56:28 atlas-oss1c7.ccs.ornl.gov kernel: [3798032.262176] LustreError: 47170:0:(service.c:3216:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message Aug 21 14:56:28 atlas-oss1c7.ccs.ornl.gov kernel: [3798032.484643] Lustre: 33195:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-27), not sending early reply Aug 21 14:56:28 atlas-oss1c7.ccs.ornl.gov kernel: [3798032.484644] req@ffff880ab65ca800 x1476723491501892/t0(0) o3->5045d212-2197-e788-761a-659b0b8106e1@3508@gni109:0/0 lens 448/432 e 0 to 0 dl 1408647393 ref 2 fl Interpret:/0/0 rc 0/0 Aug 21 14:56:28 atlas-oss1c7.ccs.ornl.gov kernel: [3798032.571578] Lustre: 33195:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 91 previous similar messages Aug 21 14:56:33 atlas-oss1c7.ccs.ornl.gov kernel: [3798037.630511] LustreError: 33142:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 0+0s req@ffff880c0f16b000 x1476723479323177/t0(0) o3->31db7be7-95d7-4210-1973-dacb1cda2d20@1052@gni109:0/0 lens 448/432 e 0 to 0 dl 1408647393 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:56:33 atlas-oss1c7.ccs.ornl.gov kernel: [3798037.669594] Lustre: atlas1-OST0256: Bulk IO read error with a286ded8-37b2-1833-f5ec-61648a7b8697 (at 3555@gni109), client will retry: rc -110 Aug 21 14:56:33 atlas-oss1c7.ccs.ornl.gov kernel: [3798037.669596] Lustre: Skipped 11 previous similar messages Aug 21 14:56:33 atlas-oss1c7.ccs.ornl.gov kernel: [3798037.764430] LustreError: 33142:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 12 previous similar messages Aug 21 14:56:37 atlas-oss1c7.ccs.ornl.gov kernel: [3798041.961536] Lustre: 33331:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (632:4s); client may timeout. req@ffff880c0f16b400 x1476723498156387/t0(0) o3->74236b16-7dd1-5fd6-1835-d68c46c1244d@8655@gni111:0/0 lens 448/432 e 0 to 0 dl 1408647393 ref 1 fl Complete:/0/ffffffff rc 0/-1 Aug 21 14:56:37 atlas-oss1c7.ccs.ornl.gov kernel: [3798042.046105] Lustre: 33331:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 10 previous similar messages Aug 21 14:57:16 atlas-oss1c7.ccs.ornl.gov kernel: [3798080.931418] LustreError: 33112:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-11429@gni111: deadline 606:38s ago Aug 21 14:57:16 atlas-oss1c7.ccs.ornl.gov kernel: [3798080.931420] req@ffff880ef51f1000 x1476723485247578/t0(0) o4->30ece88a-0205-807f-885f-0b25c5e39c74@11429@gni111:0/0 lens 448/0 e 0 to 0 dl 1408647398 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Aug 21 14:57:17 atlas-oss1c7.ccs.ornl.gov kernel: [3798081.030107] Lustre: 33290:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (635:8s); client may timeout. req@ffff88035aa15800 x1476723491669816/t0(0) o3->5b78a0dd-0aae-ae3e-beb3-8a25c0bc7b08@8777@gni108:0/0 lens 448/0 e 0 to 0 dl 1408647428 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Aug 21 14:57:17 atlas-oss1c7.ccs.ornl.gov kernel: [3798081.031328] LustreError: 33112:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 36 previous similar messages Aug 21 14:57:17 atlas-oss1c7.ccs.ornl.gov kernel: [3798081.097894] LustreError: 13583:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -3+3s req@ffff8809c12d4c00 x1476723491630127/t0(0) o3->aa9082a5-9d36-9c6e-b1ab-a1572b9c32e1@7121@gni111:0/0 lens 448/432 e 0 to 0 dl 1408647433 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:57:17 atlas-oss1c7.ccs.ornl.gov kernel: [3798081.097898] LustreError: 13583:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 13 previous similar messages Aug 21 14:57:17 atlas-oss1c7.ccs.ornl.gov kernel: [3798081.270987] Lustre: 33290:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 24 previous similar messages Aug 21 14:57:17 atlas-oss1c7.ccs.ornl.gov kernel: [3798081.310145] Lustre: atlas1-OST0136: Bulk IO read error with aa9082a5-9d36-9c6e-b1ab-a1572b9c32e1 (at 7121@gni111), client will retry: rc -110 Aug 21 14:57:17 atlas-oss1c7.ccs.ornl.gov kernel: [3798081.350494] Lustre: Skipped 16 previous similar messages Aug 21 14:57:28 atlas-oss1c7.ccs.ornl.gov kernel: [3798092.443260] LustreError: 47848:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 621s Aug 21 14:57:28 atlas-oss1c7.ccs.ornl.gov kernel: [3798092.475208] LustreError: 47848:0:(service.c:3216:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message Aug 21 14:58:08 atlas-oss1c7.ccs.ornl.gov kernel: [3798132.997614] LustreError: 33167:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 14:58:08 atlas-oss1c7.ccs.ornl.gov kernel: [3798132.997616] ns: filter-atlas1-OST00a6_UUID lock: ffff8806dfb9b900/0xecd0a12120f18d4b lrc: 4/0,0 mode: PW/PW res: [0x611d32:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 1011712->1015807) flags: 0x10020 nid: 10.36.205.198@o2ib remote: 0xbf5f421e3f598392 expref: 392 pid: 14234 timeout: 8091741909 lvb_type: 0 Aug 21 14:58:12 atlas-oss1c7.ccs.ornl.gov kernel: [3798136.524803] Lustre: 33147:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/0), not sending early reply Aug 21 14:58:12 atlas-oss1c7.ccs.ornl.gov kernel: [3798136.524805] req@ffff8801a7f69800 x1476723486315963/t0(0) o4->fe4a4c1d-6120-2c2b-e1e8-00574ee9dcda@6312@gni106:0/0 lens 448/0 e 0 to 0 dl 1408647497 ref 2 fl New:/0/ffffffff rc 0/-1 Aug 21 14:58:12 atlas-oss1c7.ccs.ornl.gov kernel: [3798136.611257] Lustre: 33147:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Aug 21 14:58:27 atlas-oss1c7.ccs.ornl.gov kernel: [3798151.754347] LustreError: 48805:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 611s Aug 21 14:58:27 atlas-oss1c7.ccs.ornl.gov kernel: [3798151.786831] LustreError: 48805:0:(service.c:3216:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message Aug 21 14:58:30 atlas-oss1c7.ccs.ornl.gov kernel: [3798154.681061] Lustre: 33385:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (605:13s); client may timeout. req@ffff8801a7f69800 x1476723486315963/t111669220737(0) o4->fe4a4c1d-6120-2c2b-e1e8-00574ee9dcda@6312@gni106:0/0 lens 448/416 e 0 to 0 dl 1408647497 ref 1 fl Complete:/0/0 rc 0/0 Aug 21 14:58:30 atlas-oss1c7.ccs.ornl.gov kernel: [3798154.768126] Lustre: 33385:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 15 previous similar messages Aug 21 14:58:42 atlas-oss1c7.ccs.ornl.gov kernel: [3798166.817087] Lustre: atlas1-OST0136: Client 995293b6-49ae-5653-9559-baf2afd203ba (at 1093@gni109) reconnecting Aug 21 14:58:42 atlas-oss1c7.ccs.ornl.gov kernel: [3798166.842395] Lustre: Skipped 5 previous similar messages Aug 21 14:58:46 atlas-oss1c7.ccs.ornl.gov kernel: [3798170.344842] Lustre: atlas1-OST0256: Client 63ba583c-83f2-5296-ff3e-fbe230564e4c (at 2573@gni109) refused reconnection, still busy with 4 active RPCs Aug 21 14:58:46 atlas-oss1c7.ccs.ornl.gov kernel: [3798170.384033] Lustre: Skipped 10 previous similar messages Aug 21 14:58:52 atlas-oss1c7.ccs.ornl.gov kernel: [3798176.195274] LustreError: 33193:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880f8a600400 x1476723486544869/t0(0) o3->d75b67ea-a112-8812-7a7b-1ddab0003ce2@2572@gni109:0/0 lens 448/432 e 0 to 0 dl 1408647626 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:58:52 atlas-oss1c7.ccs.ornl.gov kernel: [3798176.195911] LustreError: 33037:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8804e9cd5800 x1476723482167817/t0(0) o3->0e3b2556-b1ad-c609-f0b5-edc7fb12d49c@9645@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647614 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:58:52 atlas-oss1c7.ccs.ornl.gov kernel: [3798176.195976] Lustre: atlas1-OST0256: Bulk IO read error with 0e3b2556-b1ad-c609-f0b5-edc7fb12d49c (at 9645@gni108), client will retry: rc -107 Aug 21 14:58:52 atlas-oss1c7.ccs.ornl.gov kernel: [3798176.195978] Lustre: Skipped 12 previous similar messages Aug 21 14:58:52 atlas-oss1c7.ccs.ornl.gov kernel: [3798176.406661] LustreError: 33193:0:(ldlm_lib.c:2730:target_bulk_io()) Skipped 11 previous similar messages Aug 21 14:59:00 atlas-oss1c7.ccs.ornl.gov kernel: [3798184.853610] LustreError: 33299:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff88036f525400 x1476723478014588/t0(0) o3->f2a49d46-edaa-2078-aa90-efea68f7437b@8730@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647626 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:59:00 atlas-oss1c7.ccs.ornl.gov kernel: [3798184.920106] LustreError: 33299:0:(ldlm_lib.c:2730:target_bulk_io()) Skipped 13 previous similar messages Aug 21 14:59:05 atlas-oss1c7.ccs.ornl.gov kernel: [3798189.248543] LustreError: 33167:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880e88d91400 x1476723491636378/t0(0) o3->17472fb8-e81e-2116-c34e-f214f027e93c@8642@gni111:0/0 lens 448/432 e 0 to 0 dl 1408647617 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:59:05 atlas-oss1c7.ccs.ornl.gov kernel: [3798189.321734] LustreError: 33167:0:(ldlm_lib.c:2730:target_bulk_io()) Skipped 5 previous similar messages Aug 21 14:59:09 atlas-oss1c7.ccs.ornl.gov kernel: [3798193.522538] LustreError: 13582:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880785d92000 x1476723482167818/t0(0) o3->0e3b2556-b1ad-c609-f0b5-edc7fb12d49c@9645@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647618 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:59:09 atlas-oss1c7.ccs.ornl.gov kernel: [3798193.593501] LustreError: 13582:0:(ldlm_lib.c:2730:target_bulk_io()) Skipped 1 previous similar message Aug 21 14:59:13 atlas-oss1c7.ccs.ornl.gov kernel: [3798197.991778] LustreError: 33374:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880415d35c00 x1476723493760471/t0(0) o3->d9e673ce-84a8-2169-e484-045efe29cd18@7189@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647631 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:59:13 atlas-oss1c7.ccs.ornl.gov kernel: [3798198.064947] LustreError: 33374:0:(ldlm_lib.c:2730:target_bulk_io()) Skipped 1 previous similar message Aug 21 14:59:22 atlas-oss1c7.ccs.ornl.gov kernel: [3798206.507880] LustreError: 33369:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880ac92f2000 x1476723496874036/t0(0) o3->9f93c941-be9b-f3d2-30d9-863410825aec@9768@gni111:0/0 lens 448/432 e 0 to 0 dl 1408647657 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:59:22 atlas-oss1c7.ccs.ornl.gov kernel: [3798206.578460] LustreError: 33369:0:(ldlm_lib.c:2730:target_bulk_io()) Skipped 26 previous similar messages Aug 21 14:59:48 atlas-oss1c7.ccs.ornl.gov kernel: [3798232.487663] LustreError: 33370:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff88062dca5000 x1476723482153324/t0(0) o3->727e5e80-218d-83a9-7b6b-9ed56ad77584@9647@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647757 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 14:59:48 atlas-oss1c7.ccs.ornl.gov kernel: [3798232.558228] LustreError: 33370:0:(ldlm_lib.c:2730:target_bulk_io()) Skipped 29 previous similar messages Aug 21 15:00:23 atlas-oss1c7.ccs.ornl.gov kernel: [3798267.545257] LustreError: 137-5: atlas1-OST01c2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target) Aug 21 15:00:23 atlas-oss1c7.ccs.ornl.gov kernel: [3798267.571085] LustreError: Skipped 4 previous similar messages Aug 21 15:00:40 atlas-oss1c7.ccs.ornl.gov kernel: [3798284.577800] LustreError: 33145:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 15:00:40 atlas-oss1c7.ccs.ornl.gov kernel: [3798284.577802] ns: filter-atlas1-OST00a6_UUID lock: ffff880fe3f9dd80/0xecd0a12120f1aec3 lrc: 3/0,0 mode: PR/PR res: [0x611d7a:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.36.205.218@o2ib remote: 0xd9dee22077dec22b expref: 257 pid: 17903 timeout: 8091891189 lvb_type: 1 Aug 21 15:00:40 atlas-oss1c7.ccs.ornl.gov kernel: [3798284.707239] LustreError: 33145:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) Skipped 2 previous similar messages Aug 21 15:00:43 atlas-oss1c7.ccs.ornl.gov kernel: [3798287.629806] Lustre: atlas1-OST0256: Client a286ded8-37b2-1833-f5ec-61648a7b8697 (at 3555@gni109) refused reconnection, still busy with 1 active RPCs Aug 21 15:00:43 atlas-oss1c7.ccs.ornl.gov kernel: [3798287.668343] Lustre: Skipped 17 previous similar messages Aug 21 15:00:44 atlas-oss1c7.ccs.ornl.gov kernel: [3798288.863782] LustreError: 33280:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880ca1aec800 x1476723480937048/t0(0) o3->1f4e3f57-b114-a686-31c3-256b039bc29c@10048@gni111:0/0 lens 448/432 e 0 to 0 dl 1408647842 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 15:00:44 atlas-oss1c7.ccs.ornl.gov kernel: [3798288.929711] LustreError: 33280:0:(ldlm_lib.c:2730:target_bulk_io()) Skipped 14 previous similar messages Aug 21 15:00:56 atlas-oss1c7.ccs.ornl.gov kernel: [3798300.997444] LustreError: 33120:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 15:00:56 atlas-oss1c7.ccs.ornl.gov kernel: [3798300.997446] ns: filter-atlas1-OST0136_UUID lock: ffff880b9cb44d80/0xecd0a12120f1b03d lrc: 3/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 4 type: EXT [0->18446744073709551615] (req 160305152->160432127) flags: 0x20 nid: 3252@gni106 remote: 0x3d3a0f23a61278c7 expref: 5 pid: 15482 timeout: 8091909854 lvb_type: 0 Aug 21 15:01:06 atlas-oss1c7.ccs.ornl.gov kernel: [3798310.427873] LustreError: 33270:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 15:01:06 atlas-oss1c7.ccs.ornl.gov kernel: [3798310.427875] ns: filter-atlas1-OST0136_UUID lock: ffff8801b2c64d80/0xecd0a12120f1b0fa lrc: 3/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 193 type: EXT [182341632->183132159] (req 182341632->182452223) flags: 0x20 nid: 3356@gni106 remote: 0x15d191dac3c4c160 expref: 6 pid: 14267 timeout: 8091919281 lvb_type: 0 Aug 21 15:01:06 atlas-oss1c7.ccs.ornl.gov kernel: [3798310.556466] LustreError: 33270:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) Skipped 1 previous similar message Aug 21 15:01:06 atlas-oss1c7.ccs.ornl.gov kernel: [3798310.587431] LustreError: 33364:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) ### Adding a lock, but the front position is scheduled in 755 seconds Aug 21 15:01:06 atlas-oss1c7.ccs.ornl.gov kernel: [3798310.587433] ns: filter-atlas1-OST0136_UUID lock: ffff8801b2c64d80/0xecd0a12120f1b0fa lrc: 3/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 197 type: EXT [182341632->183132159] (req 182341632->182452223) flags: 0x20 nid: 3356@gni106 remote: 0x15d191dac3c4c160 expref: 6 pid: 14267 timeout: 8092299441 lvb_type: 0 Aug 21 15:01:06 atlas-oss1c7.ccs.ornl.gov kernel: [3798310.587606] Lustre: atlas1-OST0256: Bulk IO read error with 17472fb8-e81e-2116-c34e-f214f027e93c (at 8642@gni111), client will retry: rc -107 Aug 21 15:01:06 atlas-oss1c7.ccs.ornl.gov kernel: [3798310.587608] Lustre: Skipped 124 previous similar messages Aug 21 15:01:06 atlas-oss1c7.ccs.ornl.gov kernel: [3798310.787638] LustreError: 33364:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) Skipped 8 previous similar messages Aug 21 15:01:10 atlas-oss1c7.ccs.ornl.gov kernel: [3798314.759356] LustreError: 33308:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 15:01:10 atlas-oss1c7.ccs.ornl.gov kernel: [3798314.759357] ns: filter-atlas1-OST0136_UUID lock: ffff880a59bf6240/0xecd0a12120f1b108 lrc: 3/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 216 type: EXT [159518720->161091583] (req 159518720->160309247) flags: 0x20 nid: 3252@gni106 remote: 0x3d3a0f23a61278ff expref: 5 pid: 15500 timeout: 8092299682 lvb_type: 0 Aug 21 15:01:10 atlas-oss1c7.ccs.ornl.gov kernel: [3798314.888080] LustreError: 33308:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) Skipped 37 previous similar messages Aug 21 15:01:10 atlas-oss1c7.ccs.ornl.gov kernel: [3798314.919203] LustreError: 33308:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) ### Adding a lock, but the front position is scheduled in 750 seconds Aug 21 15:01:10 atlas-oss1c7.ccs.ornl.gov kernel: [3798314.919204] ns: filter-atlas1-OST0136_UUID lock: ffff8801b2c64d80/0xecd0a12120f1b0fa lrc: 3/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 216 type: EXT [182341632->183132159] (req 182341632->182452223) flags: 0x20 nid: 3356@gni106 remote: 0x15d191dac3c4c160 expref: 8 pid: 14267 timeout: 8092299441 lvb_type: 0 Aug 21 15:01:10 atlas-oss1c7.ccs.ornl.gov kernel: [3798315.058741] LustreError: 33308:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) Skipped 63 previous similar messages Aug 21 15:01:14 atlas-oss1c7.ccs.ornl.gov kernel: [3798319.090138] LustreError: 33178:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 15:01:14 atlas-oss1c7.ccs.ornl.gov kernel: [3798319.090139] ns: filter-atlas1-OST0136_UUID lock: ffff8809947d0000/0xecd0a12120f1b300 lrc: 3/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 217 type: EXT [191889408->193363967] (req 191889408->192581631) flags: 0x20 nid: 2862@gni106 remote: 0x89e580ba58bdf577 expref: 5 pid: 15500 timeout: 8091919773 lvb_type: 0 Aug 21 15:01:14 atlas-oss1c7.ccs.ornl.gov kernel: [3798319.219911] LustreError: 33178:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) Skipped 1 previous similar message Aug 21 15:01:18 atlas-oss1c7.ccs.ornl.gov kernel: [3798323.037459] LustreError: 33273:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) ### Adding a lock, but the front position is scheduled in 751 seconds Aug 21 15:01:18 atlas-oss1c7.ccs.ornl.gov kernel: [3798323.037461] ns: filter-atlas1-OST0136_UUID lock: ffff880dccf44480/0xecd0a12120f1b0c2 lrc: 3/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 250 type: EXT [163577856->165027839] (req 163577856->164245503) flags: 0x20 nid: 3250@gni106 remote: 0x197e401c30a70b7a expref: 7 pid: 17903 timeout: 8092308281 lvb_type: 0 Aug 21 15:01:18 atlas-oss1c7.ccs.ornl.gov kernel: [3798323.162330] LustreError: 33273:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) Skipped 45 previous similar messages Aug 21 15:01:23 atlas-oss1c7.ccs.ornl.gov kernel: [3798327.749295] LustreError: 33280:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 15:01:23 atlas-oss1c7.ccs.ornl.gov kernel: [3798327.749297] ns: filter-atlas1-OST0136_UUID lock: ffff880389717480/0xecd0a12120f1b19b lrc: 3/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 253 type: EXT [154796032->155582463] (req 154796032->155189247) flags: 0x20 nid: 1463@gni103 remote: 0x2de786bcda29d575 expref: 5 pid: 14195 timeout: 8091932337 lvb_type: 0 Aug 21 15:01:23 atlas-oss1c7.ccs.ornl.gov kernel: [3798327.864526] LustreError: 33280:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) Skipped 45 previous similar messages Aug 21 15:01:49 atlas-oss1c7.ccs.ornl.gov kernel: [3798353.833621] LustreError: 33271:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8804ad55d000 x1476723487404946/t0(0) o3->8b64889c-7e68-eb06-e71c-272eb805cef2@8105@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647927 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 15:01:49 atlas-oss1c7.ccs.ornl.gov kernel: [3798353.904517] LustreError: 33271:0:(ldlm_lib.c:2730:target_bulk_io()) Skipped 22 previous similar messages Aug 21 15:02:28 atlas-oss1c7.ccs.ornl.gov kernel: [3798392.701443] LustreError: 33098:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 15:02:28 atlas-oss1c7.ccs.ornl.gov kernel: [3798392.701444] ns: filter-atlas1-OST0376_UUID lock: ffff8808f34366c0/0xecd0a12120f1b02f lrc: 4/0,0 mode: PW/PW res: [0x5c2a68:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x10020 nid: 10.36.205.218@o2ib remote: 0xd9dee22077df83ed expref: 221 pid: 17903 timeout: 8091989485 lvb_type: 0 Aug 21 15:02:28 atlas-oss1c7.ccs.ornl.gov kernel: [3798392.827720] LustreError: 33098:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) Skipped 20 previous similar messages Aug 21 15:03:50 atlas-oss1c7.ccs.ornl.gov kernel: [3798474.967421] LustreError: 33328:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 15:03:50 atlas-oss1c7.ccs.ornl.gov kernel: [3798474.967423] ns: filter-atlas1-OST00a6_UUID lock: ffff880e134fb6c0/0xecd0a12120f1b6c6 lrc: 4/0,0 mode: PW/PW res: [0x611d84:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) flags: 0x10020 nid: 9311@gni102 remote: 0x573b3d9357220803 expref: 16 pid: 17903 timeout: 8092083641 lvb_type: 0 Aug 21 15:03:50 atlas-oss1c7.ccs.ornl.gov kernel: [3798475.089814] LustreError: 33328:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) Skipped 3 previous similar messages Aug 21 15:03:59 atlas-oss1c7.ccs.ornl.gov kernel: [3798483.630348] LustreError: 33177:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880beb614400 x1476723496878097/t0(0) o3->9f93c941-be9b-f3d2-30d9-863410825aec@9768@gni111:0/0 lens 448/432 e 0 to 0 dl 1408648087 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 15:03:59 atlas-oss1c7.ccs.ornl.gov kernel: [3798483.703196] LustreError: 33177:0:(ldlm_lib.c:2730:target_bulk_io()) Skipped 99 previous similar messages Aug 21 15:05:32 atlas-oss1c7.ccs.ornl.gov kernel: [3798576.473296] LustreError: 51598:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 611s Aug 21 15:05:32 atlas-oss1c7.ccs.ornl.gov kernel: [3798576.498933] LustreError: 51598:0:(service.c:3216:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message Aug 21 15:05:38 atlas-oss1c7.ccs.ornl.gov kernel: [3798583.226607] Lustre: atlas1-OST0256: Bulk IO read error with 0e3b2556-b1ad-c609-f0b5-edc7fb12d49c (at 9645@gni108), client will retry: rc -107 Aug 21 15:05:38 atlas-oss1c7.ccs.ornl.gov kernel: [3798583.260485] Lustre: Skipped 156 previous similar messages Aug 21 15:05:43 atlas-oss1c7.ccs.ornl.gov kernel: [3798587.555400] LustreError: 33174:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 15:05:43 atlas-oss1c7.ccs.ornl.gov kernel: [3798587.555402] ns: filter-atlas1-OST0016_UUID lock: ffff880965818d80/0xecd0a12120f187a2 lrc: 4/0,0 mode: PW/PW res: [0x562849:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 831@gni112 remote: 0xf31b2de74cbfbee2 expref: 6 pid: 17898 timeout: 8092186712 lvb_type: 0 Aug 21 15:06:01 atlas-oss1c7.ccs.ornl.gov kernel: [3798605.698606] Lustre: 33103:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-27), not sending early reply Aug 21 15:06:01 atlas-oss1c7.ccs.ornl.gov kernel: [3798605.698608] req@ffff88092be8c000 x1476723486810637/t0(0) o4->29a14412-161b-27cb-3da7-e2bf7cff9928@1184@gni109:0/0 lens 448/0 e 0 to 0 dl 1408647966 ref 2 fl New:/0/ffffffff rc 0/-1 Aug 21 15:06:01 atlas-oss1c7.ccs.ornl.gov kernel: [3798605.789199] Lustre: 33103:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Aug 21 15:06:17 atlas-oss1c7.ccs.ornl.gov kernel: [3798622.195382] LustreError: 33030:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-1184@gni109: deadline 632:11s ago Aug 21 15:06:17 atlas-oss1c7.ccs.ornl.gov kernel: [3798622.195384] req@ffff88092be8c000 x1476723486810637/t0(0) o4->29a14412-161b-27cb-3da7-e2bf7cff9928@1184@gni109:0/0 lens 448/0 e 0 to 0 dl 1408647966 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Aug 21 15:06:17 atlas-oss1c7.ccs.ornl.gov kernel: [3798622.295318] LustreError: 33030:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 16 previous similar messages Aug 21 15:06:17 atlas-oss1c7.ccs.ornl.gov kernel: [3798622.326303] Lustre: 33030:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (632:11s); client may timeout. req@ffff88092be8c000 x1476723486810637/t0(0) o4->29a14412-161b-27cb-3da7-e2bf7cff9928@1184@gni109:0/0 lens 448/0 e 0 to 0 dl 1408647966 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Aug 21 15:06:32 atlas-oss1c7.ccs.ornl.gov kernel: [3798636.537811] LustreError: 52023:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 620s Aug 21 15:06:32 atlas-oss1c7.ccs.ornl.gov kernel: [3798636.570279] LustreError: 52023:0:(service.c:3216:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message Aug 21 15:08:32 atlas-oss1c7.ccs.ornl.gov kernel: [3798757.135724] LustreError: 53516:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 619s Aug 21 15:08:32 atlas-oss1c7.ccs.ornl.gov kernel: [3798757.167112] LustreError: 53516:0:(service.c:3216:ptlrpc_svcpt_health_check()) Skipped 3 previous similar messages Aug 21 15:08:49 atlas-oss1c7.ccs.ornl.gov kernel: [3798773.751847] LustreError: 33232:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -3+3s req@ffff880c9660d400 x1476723490521461/t0(0) o3->b948fe70-9956-5245-6e02-4aea7e12d695@8252@gni111:0/0 lens 448/432 e 0 to 0 dl 1408648126 ref 1 fl Interpret:/2/0 rc 0/0 Aug 21 15:08:49 atlas-oss1c7.ccs.ornl.gov kernel: [3798773.752097] LustreError: 33194:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-971@gni112: deadline 601:2s ago Aug 21 15:08:49 atlas-oss1c7.ccs.ornl.gov kernel: [3798773.752099] req@ffff880afd6b6400 x1476723493744500/t0(0) o3->dd483d51-77cc-7995-c7d8-21b018d6ef25@971@gni112:0/0 lens 448/0 e 0 to 0 dl 1408648127 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Aug 21 15:08:49 atlas-oss1c7.ccs.ornl.gov kernel: [3798773.942546] LustreError: 33232:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 15 previous similar messages Aug 21 15:08:58 atlas-oss1c7.ccs.ornl.gov kernel: [3798782.413117] LustreError: 33385:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -11+11s req@ffff8809c598d400 x1476723507481225/t0(0) o3->522b4917-96ac-21dd-44e4-8a8222a66ecf@11493@gni102:0/0 lens 448/432 e 0 to 0 dl 1408648126 ref 1 fl Interpret:/2/0 rc 0/0 Aug 21 15:08:58 atlas-oss1c7.ccs.ornl.gov kernel: [3798782.414378] LustreError: 33213:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10053@gni111: deadline 600:7s ago Aug 21 15:08:58 atlas-oss1c7.ccs.ornl.gov kernel: [3798782.414379] req@ffff8809e55ec000 x1476723487484224/t0(0) o3->4bd334e7-2d26-b073-0ceb-ddb194a77bfc@10053@gni111:0/0 lens 448/0 e 0 to 0 dl 1408648130 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Aug 21 15:08:58 atlas-oss1c7.ccs.ornl.gov kernel: [3798782.414383] LustreError: 33213:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 197 previous similar messages Aug 21 15:08:58 atlas-oss1c7.ccs.ornl.gov kernel: [3798782.626990] LustreError: 33385:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 9 previous similar messages Aug 21 15:09:15 atlas-oss1c7.ccs.ornl.gov kernel: [3798799.730662] LustreError: 33087:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -25+25s req@ffff880beb9c3c00 x1476723490525369/t0(0) o3->165150a5-cbd8-acfe-cbbb-9b45e1193c49@8646@gni111:0/0 lens 448/432 e 0 to 0 dl 1408648130 ref 1 fl Interpret:/2/0 rc 0/0 Aug 21 15:09:15 atlas-oss1c7.ccs.ornl.gov kernel: [3798799.814346] LustreError: 33087:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 98 previous similar messages Aug 21 15:09:15 atlas-oss1c7.ccs.ornl.gov kernel: [3798799.859920] LustreError: 33250:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-8176@gni108: deadline 600:12s ago Aug 21 15:09:15 atlas-oss1c7.ccs.ornl.gov kernel: [3798799.859922] req@ffff88047bb2cc00 x1476723477005813/t0(0) o3->efe82b3b-0260-bcd7-82e7-b8907255092c@8176@gni108:0/0 lens 448/0 e 0 to 0 dl 1408648143 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Aug 21 15:09:15 atlas-oss1c7.ccs.ornl.gov kernel: [3798799.953515] LustreError: 33250:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 446 previous similar messages Aug 21 15:09:42 atlas-oss1c7.ccs.ornl.gov kernel: [3798827.341252] LustreError: 33219:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 15:09:42 atlas-oss1c7.ccs.ornl.gov kernel: [3798827.341254] ns: filter-atlas1-OST00a6_UUID lock: ffff880c0d1ffd80/0xecd0a12120f1c115 lrc: 4/0,0 mode: PW/PW res: [0x611d92:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x10020 nid: 10.36.202.130@o2ib remote: 0x3d25f9173372574f expref: 5 pid: 17898 timeout: 8092435999 lvb_type: 0 Aug 21 15:09:45 atlas-oss1c7.ccs.ornl.gov kernel: [3798830.203425] LustreError: 33342:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880b9c665000 x1476723491513378/t0(0) o3->5045d212-2197-e788-761a-659b0b8106e1@3508@gni109:0/0 lens 448/432 e 0 to 0 dl 1408648308 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 15:09:45 atlas-oss1c7.ccs.ornl.gov kernel: [3798830.274939] LustreError: 33342:0:(ldlm_lib.c:2730:target_bulk_io()) Skipped 88 previous similar messages Aug 21 15:09:49 atlas-oss1c7.ccs.ornl.gov kernel: [3798833.801034] Lustre: atlas1-OST02e6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting Aug 21 15:09:49 atlas-oss1c7.ccs.ornl.gov kernel: [3798833.825888] Lustre: Skipped 224 previous similar messages Aug 21 15:09:54 atlas-oss1c7.ccs.ornl.gov kernel: [3798838.748040] LustreError: 33138:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-8105@gni108: deadline 600:24s ago Aug 21 15:09:54 atlas-oss1c7.ccs.ornl.gov kernel: [3798838.748042] req@ffff8800700e2c00 x1476723487427003/t0(0) o3->8b64889c-7e68-eb06-e71c-272eb805cef2@8105@gni108:0/0 lens 448/0 e 0 to 0 dl 1408648170 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Aug 21 15:09:54 atlas-oss1c7.ccs.ornl.gov kernel: [3798838.847367] LustreError: 33138:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 152 previous similar messages Aug 21 15:09:54 atlas-oss1c7.ccs.ornl.gov kernel: [3798839.400066] Lustre: atlas1-OST0256: Client 5045d212-2197-e788-761a-659b0b8106e1 (at 3508@gni109) refused reconnection, still busy with 2 active RPCs Aug 21 15:09:54 atlas-oss1c7.ccs.ornl.gov kernel: [3798839.437745] Lustre: Skipped 2 previous similar messages Aug 21 15:10:49 atlas-oss1c7.ccs.ornl.gov kernel: [3798893.807136] Lustre: 33171:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-1), not sending early reply Aug 21 15:10:49 atlas-oss1c7.ccs.ornl.gov kernel: [3798893.807138] req@ffff880e18f41c00 x1476723485261301/t0(0) o4->30ece88a-0205-807f-885f-0b25c5e39c74@11429@gni111:0/0 lens 448/0 e 0 to 0 dl 1408648254 ref 2 fl New:/2/ffffffff rc 0/-1 Aug 21 15:10:49 atlas-oss1c7.ccs.ornl.gov kernel: [3798893.898473] Lustre: 33171:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 17 previous similar messages Aug 21 15:11:21 atlas-oss1c7.ccs.ornl.gov kernel: [3798925.320725] LustreError: 33358:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-8655@gni111: deadline 600:37s ago Aug 21 15:11:21 atlas-oss1c7.ccs.ornl.gov kernel: [3798925.320727] req@ffff880c9f504c00 x1476723498168614/t0(0) o3->74236b16-7dd1-5fd6-1835-d68c46c1244d@8655@gni111:0/0 lens 448/0 e 0 to 0 dl 1408648243 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Aug 21 15:11:21 atlas-oss1c7.ccs.ornl.gov kernel: [3798925.321869] Lustre: 33365:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:37s); client may timeout. req@ffff880e18f41400 x1476723498168615/t0(0) o3->74236b16-7dd1-5fd6-1835-d68c46c1244d@8655@gni111:0/0 lens 448/0 e 0 to 0 dl 1408648243 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Aug 21 15:11:21 atlas-oss1c7.ccs.ornl.gov kernel: [3798925.321873] Lustre: 33365:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 1038 previous similar messages Aug 21 15:11:21 atlas-oss1c7.ccs.ornl.gov kernel: [3798925.550096] LustreError: 33358:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 22 previous similar messages Aug 21 15:11:38 atlas-oss1c7.ccs.ornl.gov kernel: [3798943.293821] LustreError: 54175:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 607s Aug 21 15:11:38 atlas-oss1c7.ccs.ornl.gov kernel: [3798943.317662] LustreError: 54175:0:(service.c:3216:ptlrpc_svcpt_health_check()) Skipped 5 previous similar messages Aug 21 15:16:06 atlas-oss1c7.ccs.ornl.gov kernel: [3799211.095669] LustreError: 33296:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 15:16:06 atlas-oss1c7.ccs.ornl.gov kernel: [3799211.095671] ns: filter-atlas1-OST0376_UUID lock: ffff880da4549240/0xecd0a12120f1c5bb lrc: 3/0,0 mode: PR/PR res: [0x5c2a95:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.36.205.218@o2ib remote: 0xd9dee22077e8a32a expref: 239 pid: 17872 timeout: 8092818585 lvb_type: 1 Aug 21 15:16:06 atlas-oss1c7.ccs.ornl.gov kernel: [3799211.219582] LustreError: 33296:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) Skipped 284 previous similar messages Aug 21 15:16:23 atlas-oss1c7.ccs.ornl.gov kernel: [3799227.601506] LustreError: 14436:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) ### Adding a lock, but the front position is scheduled in 742 seconds Aug 21 15:16:23 atlas-oss1c7.ccs.ornl.gov kernel: [3799227.601508] ns: filter-atlas1-OST0256_UUID lock: ffff88031c052b40/0xecd0a12120f154b8 lrc: 3/0,0 mode: PW/PW res: [0x598e59:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 nid: 12784@gni107 remote: 0xaf85df49696147fb expref: 94 pid: 14207 timeout: 8093203930 lvb_type: 0 Aug 21 15:16:23 atlas-oss1c7.ccs.ornl.gov kernel: [3799227.725361] LustreError: 14436:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) Skipped 33 previous similar messages Aug 21 15:16:32 atlas-oss1c7.ccs.ornl.gov kernel: [3799236.904143] LustreError: 14196:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) ### Adding a lock, but the front position is scheduled in 733 seconds Aug 21 15:16:32 atlas-oss1c7.ccs.ornl.gov kernel: [3799236.904145] ns: filter-atlas1-OST0256_UUID lock: ffff88031c052b40/0xecd0a12120f154b8 lrc: 3/0,0 mode: PW/PW res: [0x598e59:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 nid: 12784@gni107 remote: 0xaf85df49696147fb expref: 94 pid: 14207 timeout: 8093203930 lvb_type: 0 Aug 21 15:16:32 atlas-oss1c7.ccs.ornl.gov kernel: [3799237.028341] LustreError: 14196:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) Skipped 1 previous similar message Aug 21 15:16:45 atlas-oss1c7.ccs.ornl.gov kernel: [3799250.121141] LustreError: 56477:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 619s Aug 21 15:16:45 atlas-oss1c7.ccs.ornl.gov kernel: [3799250.153337] LustreError: 56477:0:(service.c:3216:ptlrpc_svcpt_health_check()) Skipped 9 previous similar messages Aug 21 15:17:07 atlas-oss1c7.ccs.ornl.gov kernel: [3799271.823169] LustreError: 13591:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-9402@gni105: deadline 632:8s ago Aug 21 15:17:07 atlas-oss1c7.ccs.ornl.gov kernel: [3799271.823171] req@ffff880c0684f000 x1476723490953721/t0(0) o4->c19dc26e-ee45-5cac-5157-d0e883760321@9402@gni105:0/0 lens 448/0 e 0 to 0 dl 1408648619 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Aug 21 15:17:07 atlas-oss1c7.ccs.ornl.gov kernel: [3799271.921572] LustreError: 13591:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 57 previous similar messages Aug 21 15:17:13 atlas-oss1c7.ccs.ornl.gov kernel: [3799278.468715] LustreError: 33296:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) ### Adding a lock, but the front position is scheduled in 754 seconds Aug 21 15:17:13 atlas-oss1c7.ccs.ornl.gov kernel: [3799278.468717] ns: filter-atlas1-OST02e6_UUID lock: ffff8810664d1900/0xecd0a12120f15479 lrc: 3/0,0 mode: PW/PW res: [0x55956e:0x0:0x0].0 rrc: 7 type: EXT [0->18446744073709551615] (req 0->229375) flags: 0x20 nid: 13594@gni104 remote: 0x3fc7da736221e791 expref: 49 pid: 14405 timeout: 8093266812 lvb_type: 0 Aug 21 15:17:13 atlas-oss1c7.ccs.ornl.gov kernel: [3799278.594322] LustreError: 33296:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) Skipped 5 previous similar messages Aug 21 15:19:33 atlas-oss1c7.ccs.ornl.gov kernel: [3799418.004601] Lustre: 33215:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Aug 21 15:19:33 atlas-oss1c7.ccs.ornl.gov kernel: [3799418.004603] req@ffff880b4a741c00 x1476723485575376/t0(0) o4->40e386fd-8fe6-516c-c0da-8744f7c6a18c@2121@gni112:0/0 lens 448/448 e 0 to 0 dl 1408648778 ref 2 fl Interpret:/0/0 rc 0/0 Aug 21 15:19:33 atlas-oss1c7.ccs.ornl.gov kernel: [3799418.096417] Lustre: 33215:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 7 previous similar messages Aug 21 15:19:55 atlas-oss1c7.ccs.ornl.gov kernel: [3799440.594751] Lustre: 33259:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:114s); client may timeout. req@ffff880e32f14000 x1476723486561030/t0(0) o3->d75b67ea-a112-8812-7a7b-1ddab0003ce2@2572@gni109:0/0 lens 448/0 e 0 to 0 dl 1408648681 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Aug 21 15:19:55 atlas-oss1c7.ccs.ornl.gov kernel: [3799440.686478] Lustre: 33259:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 420 previous similar messages Aug 21 15:20:39 atlas-oss1c7.ccs.ornl.gov kernel: [3799483.887475] Lustre: atlas1-OST0256: Slow creates, 128/256 objects created at a rate of 2/s Aug 21 15:20:44 atlas-oss1c7.ccs.ornl.gov kernel: [3799489.426737] LustreError: 17904:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) ### Adding a lock, but the front position is scheduled in 753 seconds Aug 21 15:20:44 atlas-oss1c7.ccs.ornl.gov kernel: [3799489.426739] ns: filter-atlas1-OST0136_UUID lock: ffff88031c052d80/0xecd0a12120f1548e lrc: 3/0,0 mode: PW/PW res: [0x5aef88:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 nid: 13954@gni101 remote: 0xe7081c32c3e5fec5 expref: 87 pid: 14207 timeout: 8093476629 lvb_type: 0 Aug 21 15:20:44 atlas-oss1c7.ccs.ornl.gov kernel: [3799489.554310] LustreError: 17904:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) Skipped 94 previous similar messages Aug 21 15:21:05 atlas-oss1c7.ccs.ornl.gov kernel: [3799509.873217] LustreError: 33291:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -6+6s req@ffff8808d5349800 x1476723499160241/t0(0) o3->1ea86b9a-13a3-c2fd-2a66-cdc91dd992db@10015@gni102:0/0 lens 448/432 e 0 to 0 dl 1408648859 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 15:21:05 atlas-oss1c7.ccs.ornl.gov kernel: [3799509.873293] Lustre: atlas1-OST0256: Bulk IO read error with 1ea86b9a-13a3-c2fd-2a66-cdc91dd992db (at 10015@gni102), client will retry: rc -110 Aug 21 15:21:05 atlas-oss1c7.ccs.ornl.gov kernel: [3799509.873296] Lustre: Skipped 222 previous similar messages Aug 21 15:21:05 atlas-oss1c7.ccs.ornl.gov kernel: [3799510.014449] LustreError: 33291:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 59 previous similar messages Aug 21 15:21:09 atlas-oss1c7.ccs.ornl.gov kernel: [3799514.203854] LustreError: 13589:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -10+10s req@ffff880de3e80c00 x1476723496266111/t0(0) o3->995293b6-49ae-5653-9559-baf2afd203ba@1093@gni109:0/0 lens 448/432 e 0 to 0 dl 1408648859 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 15:21:09 atlas-oss1c7.ccs.ornl.gov kernel: [3799514.284883] LustreError: 13589:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 35 previous similar messages Aug 21 15:21:09 atlas-oss1c7.ccs.ornl.gov kernel: [3799514.696829] Lustre: atlas1-OST01c6: Client c19dc26e-ee45-5cac-5157-d0e883760321 (at 9402@gni105) reconnecting Aug 21 15:21:09 atlas-oss1c7.ccs.ornl.gov kernel: [3799514.723668] Lustre: Skipped 76 previous similar messages Aug 21 15:21:22 atlas-oss1c7.ccs.ornl.gov kernel: [3799527.197791] LustreError: 33351:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -1+1s req@ffff8801d1f8dc00 x1476723489496475/t0(0) o3->0dc138fd-aeb6-57c8-3c03-6c1cb85e7772@7193@gni108:0/0 lens 448/432 e 0 to 0 dl 1408648881 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 15:21:22 atlas-oss1c7.ccs.ornl.gov kernel: [3799527.278909] LustreError: 33351:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 6 previous similar messages Aug 21 15:21:44 atlas-oss1c7.ccs.ornl.gov kernel: [3799549.066256] LustreError: 33384:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-8105@gni108: deadline 600:56s ago Aug 21 15:21:44 atlas-oss1c7.ccs.ornl.gov kernel: [3799549.066258] req@ffff8805c3b39c00 x1476723487437143/t0(0) o3->8b64889c-7e68-eb06-e71c-272eb805cef2@8105@gni108:0/0 lens 448/0 e 0 to 0 dl 1408648848 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Aug 21 15:21:44 atlas-oss1c7.ccs.ornl.gov kernel: [3799549.176869] LustreError: 33384:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 394 previous similar messages Aug 21 15:22:01 atlas-oss1c7.ccs.ornl.gov kernel: [3799566.207438] LustreError: 13589:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 0+0s req@ffff880d3a8fdc00 x1476723496347392/t0(0) o3->abd980c8-e206-9cec-e323-5138ec18e1c2@2090@gni112:0/0 lens 448/432 e 0 to 0 dl 1408648921 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 15:22:01 atlas-oss1c7.ccs.ornl.gov kernel: [3799566.282869] LustreError: 13589:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 44 previous similar messages Aug 21 15:22:36 atlas-oss1c7.ccs.ornl.gov kernel: [3799600.805474] LustreError: 33326:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -7+7s req@ffff880c2f503c00 x1476723489673693/t0(0) o3->70f3db2a-1574-9ff1-5542-81501e1cecc2@2504@gni112:0/0 lens 448/432 e 0 to 0 dl 1408648949 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 15:22:36 atlas-oss1c7.ccs.ornl.gov kernel: [3799600.887856] LustreError: 33326:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 67 previous similar messages Aug 21 15:22:44 atlas-oss1c7.ccs.ornl.gov kernel: [3799609.624118] LustreError: 33116:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) ### Adding a lock, but the front position is scheduled in 754 seconds Aug 21 15:22:44 atlas-oss1c7.ccs.ornl.gov kernel: [3799609.624120] ns: filter-atlas1-OST0136_UUID lock: ffff8807cf2ad000/0xecd0a12120f1b481 lrc: 3/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 191 type: EXT [14696448->15728639] (req 14696448->15486975) flags: 0x20 nid: 4390@gni106 remote: 0x93da48995802319e expref: 9 pid: 14195 timeout: 8093597952 lvb_type: 0 Aug 21 15:22:44 atlas-oss1c7.ccs.ornl.gov kernel: [3799609.750052] LustreError: 33116:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) Skipped 1 previous similar message Aug 21 15:23:36 atlas-oss1c7.ccs.ornl.gov kernel: [3799661.417391] LustreError: 33268:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) ### Adding a lock, but the front position is scheduled in 746 seconds Aug 21 15:23:36 atlas-oss1c7.ccs.ornl.gov kernel: [3799661.417393] ns: filter-atlas1-OST0136_UUID lock: ffff880101b82d80/0xecd0a12120f1b609 lrc: 3/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 199 type: EXT [181555200->182452223] (req 181555200->182345727) flags: 0x20 nid: 3356@gni106 remote: 0x15d191dac3c4c17c expref: 8 pid: 14195 timeout: 8093641116 lvb_type: 0 Aug 21 15:23:36 atlas-oss1c7.ccs.ornl.gov kernel: [3799661.549688] LustreError: 33268:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) Skipped 3 previous similar messages Aug 21 15:24:02 atlas-oss1c7.ccs.ornl.gov kernel: [3799687.416110] LustreError: 33050:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -6+6s req@ffff880b475ae800 x1476723497134565/t0(0) o3->e7728f89-024c-424f-ce0c-912ba41e98aa@1111@gni109:0/0 lens 448/432 e 0 to 0 dl 1408649036 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 15:24:02 atlas-oss1c7.ccs.ornl.gov kernel: [3799687.489417] LustreError: 33050:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 251 previous similar messages Aug 21 15:24:41 atlas-oss1c7.ccs.ornl.gov kernel: [3799726.495244] LustreError: 33208:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 15:24:41 atlas-oss1c7.ccs.ornl.gov kernel: [3799726.495245] ns: filter-atlas1-OST0136_UUID lock: ffff88042a8b5900/0xecd0a12120f1c8bd lrc: 4/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 218 type: EXT [19922944->20996095] (req 19922944->20209663) flags: 0x20 nid: 4358@gni106 remote: 0x3d2aaf2bbc025961 expref: 9 pid: 14424 timeout: 8093697429 lvb_type: 0 Aug 21 15:24:41 atlas-oss1c7.ccs.ornl.gov kernel: [3799726.614979] LustreError: 33208:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) Skipped 347 previous similar messages Aug 21 15:24:41 atlas-oss1c7.ccs.ornl.gov kernel: [3799726.654047] LustreError: 33208:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) ### Adding a lock, but the front position is scheduled in 737 seconds Aug 21 15:24:41 atlas-oss1c7.ccs.ornl.gov kernel: [3799726.654048] ns: filter-atlas1-OST0136_UUID lock: ffff8809f08e2240/0xecd0a12120f1b23c lrc: 4/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 219 type: EXT [90255360->91226111] (req 90255360->91045887) flags: 0x20 nid: 5630@gni108 remote: 0xbe37878d573f13b8 expref: 8 pid: 17904 timeout: 8093697431 lvb_type: 0 Aug 21 15:24:41 atlas-oss1c7.ccs.ornl.gov kernel: [3799726.793525] LustreError: 33208:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) Skipped 20 previous similar messages Aug 21 15:25:44 atlas-oss1c7.ccs.ornl.gov kernel: [3799789.120588] LustreError: 60474:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 733s Aug 21 15:25:44 atlas-oss1c7.ccs.ornl.gov kernel: [3799789.148757] LustreError: 60474:0:(service.c:3216:ptlrpc_svcpt_health_check()) Skipped 17 previous similar messages Aug 21 15:25:59 atlas-oss1c7.ccs.ornl.gov kernel: [3799804.332199] Lustre: atlas1-OST00a6: Bulk IO write error with 3b89af0d-64a9-10ef-9f21-1ca5c297653a (at 2721@gni109), client will retry: rc -110 Aug 21 15:26:12 atlas-oss1c7.ccs.ornl.gov kernel: [3799817.302013] LustreError: 33371:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -7+7s req@ffff8808c102a000 x1476723493419310/t0(0) o3->40486334-1d94-b0c8-38e0-ef72c93eaf6d@2505@gni112:0/0 lens 448/432 e 0 to 0 dl 1408649165 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 15:26:12 atlas-oss1c7.ccs.ornl.gov kernel: [3799817.377888] LustreError: 33371:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 758 previous similar messages Aug 21 15:26:29 atlas-oss1c7.ccs.ornl.gov kernel: [3799834.635784] Lustre: atlas1-OST0256: Bulk IO write error with 8d567e0e-a3d9-8947-7ad5-027aba01006a (at 10.36.202.138@o2ib), client will retry: rc -110 Aug 21 15:26:29 atlas-oss1c7.ccs.ornl.gov kernel: [3799834.676221] Lustre: Skipped 2 previous similar messages Aug 21 15:26:35 atlas-oss1c7.ccs.ornl.gov kernel: [3799839.991217] Lustre: atlas1-OST0256: Client 5cd81604-e92f-1636-d371-c3ed16091627 (at 11489@gni102) refused reconnection, still busy with 3 active RPCs Aug 21 15:26:38 atlas-oss1c7.ccs.ornl.gov kernel: [3799843.380472] LustreError: 33125:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880e9ed9dc00 x1476723494796402/t0(0) o3->5cd81604-e92f-1636-d371-c3ed16091627@11489@gni102:0/0 lens 448/432 e 0 to 0 dl 1408649191 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 15:26:38 atlas-oss1c7.ccs.ornl.gov kernel: [3799843.457723] LustreError: 33125:0:(ldlm_lib.c:2730:target_bulk_io()) Skipped 12 previous similar messages Aug 21 15:27:00 atlas-oss1c7.ccs.ornl.gov kernel: [3799864.936003] Lustre: atlas1-OST02e6: Bulk IO write error with eb7ff42e-b25c-a0e1-be61-4a807dad5da0 (at 6215@gni102), client will retry: rc -110 Aug 21 15:27:22 atlas-oss1c7.ccs.ornl.gov kernel: [3799887.053702] Lustre: atlas1-OST0256: Client bc33b617-6704-7e1c-881f-c890001619e4 (at 2281@gni103) refused reconnection, still busy with 2 active RPCs Aug 21 15:27:22 atlas-oss1c7.ccs.ornl.gov kernel: [3799887.095077] Lustre: Skipped 6 previous similar messages Aug 21 15:28:40 atlas-oss1c7.ccs.ornl.gov kernel: [3799965.246514] Lustre: atlas1-OST0256: Client 3326d77b-5515-0a53-39fa-660d016f10a8 (at 9654@gni108) refused reconnection, still busy with 1 active RPCs Aug 21 15:28:40 atlas-oss1c7.ccs.ornl.gov kernel: [3799965.284664] Lustre: Skipped 4 previous similar messages Aug 21 15:30:19 atlas-oss1c7.ccs.ornl.gov kernel: [3800064.327919] LustreError: 33370:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-3508@gni109: deadline 600:74s ago Aug 21 15:30:19 atlas-oss1c7.ccs.ornl.gov kernel: [3800064.327920] req@ffff880b09ae1400 x1476723491529994/t0(0) o3->5045d212-2197-e788-761a-659b0b8106e1@3508@gni109:0/0 lens 448/0 e 0 to 0 dl 1408649345 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Aug 21 15:30:19 atlas-oss1c7.ccs.ornl.gov kernel: [3800064.328649] Lustre: 33283:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:74s); client may timeout. req@ffff880b64735400 x1476723491529995/t0(0) o3->5045d212-2197-e788-761a-659b0b8106e1@3508@gni109:0/0 lens 448/0 e 0 to 0 dl 1408649345 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Aug 21 15:30:19 atlas-oss1c7.ccs.ornl.gov kernel: [3800064.328654] Lustre: 33283:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 6602 previous similar messages Aug 21 15:30:19 atlas-oss1c7.ccs.ornl.gov kernel: [3800064.552327] LustreError: 33370:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 2630 previous similar messages Aug 21 15:31:10 atlas-oss1c7.ccs.ornl.gov kernel: [3800114.970526] Lustre: atlas1-OST00a6: Client 5045d212-2197-e788-761a-659b0b8106e1 (at 3508@gni109) reconnecting Aug 21 15:31:10 atlas-oss1c7.ccs.ornl.gov kernel: [3800115.001334] Lustre: Skipped 748 previous similar messages Aug 21 15:31:11 atlas-oss1c7.ccs.ornl.gov kernel: [3800116.083178] Lustre: atlas1-OST0136: Bulk IO read error with d8f96ae7-d092-23e9-b5e7-9aec71263559 (at 1049@gni109), client will retry: rc -107 Aug 21 15:31:11 atlas-oss1c7.ccs.ornl.gov kernel: [3800116.121666] Lustre: Skipped 3861 previous similar messages Aug 21 15:31:12 atlas-oss1c7.ccs.ornl.gov kernel: [3800117.047812] Lustre: atlas1-OST0136: Client 59979f41-f3e9-1509-5d66-1344d6216769 (at 2630@gni109) refused reconnection, still busy with 2 active RPCs Aug 21 15:31:12 atlas-oss1c7.ccs.ornl.gov kernel: [3800117.082108] Lustre: Skipped 38 previous similar messages Aug 21 15:32:11 atlas-oss1c7.ccs.ornl.gov kernel: [3800176.715023] LustreError: 33358:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after -3+3s req@ffff880f786e4000 x1476723479352014/t0(0) o3->31db7be7-95d7-4210-1973-dacb1cda2d20@1052@gni109:0/0 lens 448/432 e 0 to 0 dl 1408649528 ref 1 fl Interpret:/2/0 rc 0/0 Aug 21 15:32:11 atlas-oss1c7.ccs.ornl.gov kernel: [3800176.794368] LustreError: 33358:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 2511 previous similar messages Aug 21 15:32:12 atlas-oss1c7.ccs.ornl.gov kernel: [3800177.133246] LustreError: 33167:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) ### Adding a lock, but the front position is scheduled in 754 seconds Aug 21 15:32:12 atlas-oss1c7.ccs.ornl.gov kernel: [3800177.133248] ns: filter-atlas1-OST0136_UUID lock: ffff880bb7a9b240/0xecd0a12120f1c6a2 lrc: 3/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 169 type: EXT [48541696->49328127] (req 48541696->49283071) flags: 0x20 nid: 4952@gni109 remote: 0x9d773c0a6f1f6395 expref: 7 pid: 17904 timeout: 8094165282 lvb_type: 0 Aug 21 15:32:12 atlas-oss1c7.ccs.ornl.gov kernel: [3800177.265230] LustreError: 33167:0:(ldlm_lockd.c:484:__ldlm_add_waiting_lock()) Skipped 21 previous similar messages Aug 21 15:33:09 atlas-oss1c7.ccs.ornl.gov kernel: [3800234.020585] Lustre: atlas1-OST0016: haven't heard from client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) in 1513 seconds. I think it's dead, and I am evicting it. exp ffff880998989c00, cur 1408649589 expire 1408648689 last 1408648076 Aug 21 15:33:24 atlas-oss1c7.ccs.ornl.gov kernel: [3800249.778992] LNet: Service thread pid 15469 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 21 15:33:24 atlas-oss1c7.ccs.ornl.gov kernel: [3800249.822934] Pid: 15469, comm: ll_ost03_061 Aug 21 15:33:24 atlas-oss1c7.ccs.ornl.gov kernel: [3800249.842090] Aug 21 15:33:24 atlas-oss1c7.ccs.ornl.gov kernel: [3800249.842091] Call Trace: Aug 21 15:33:24 atlas-oss1c7.ccs.ornl.gov kernel: [3800249.852653] [] ? wake_up_bit+0x2f/0x40 Aug 21 15:33:24 atlas-oss1c7.ccs.ornl.gov kernel: [3800249.872245] [] __mutex_lock_slowpath+0x13e/0x180 Aug 21 15:33:24 atlas-oss1c7.ccs.ornl.gov kernel: [3800249.892602] [] ? snprintf+0x34/0x40 Aug 21 15:33:24 atlas-oss1c7.ccs.ornl.gov kernel: [3800249.912244] [] mutex_lock+0x2b/0x50 Aug 21 15:33:24 atlas-oss1c7.ccs.ornl.gov kernel: [3800249.931873] [] osd_obj_map_lookup+0x207/0x750 [osd_ldiskfs] Aug 21 15:33:24 atlas-oss1c7.ccs.ornl.gov kernel: [3800249.952754] [] ? ldiskfs_xattr_get+0x10c/0x330 [ldiskfs] Aug 21 15:33:24 atlas-oss1c7.ccs.ornl.gov kernel: [3800249.973259] [] osd_oi_lookup+0xa7/0x140 [osd_ldiskfs] Aug 21 15:33:24 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.002345] [] osd_object_init+0x566/0xfb0 [osd_ldiskfs] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.022989] [] lu_object_alloc+0xcd/0x300 [obdclass] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.043480] [] ? htable_lookup+0x119/0x1c0 [obdclass] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.072134] [] lu_object_find_at+0x205/0x360 [obdclass] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.092700] [] ? ofd_key_init+0x59/0x1a0 [ofd] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.113083] [] ? keys_fill+0x6f/0x190 [obdclass] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.133457] [] lu_object_find+0x16/0x20 [obdclass] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.153568] [] ofd_object_find+0x35/0xf0 [ofd] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.181986] [] ? lu_env_init+0x1e/0x30 [obdclass] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.202198] [] ofd_lvbo_update+0x6d9/0xea8 [ofd] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.222610] [] ldlm_request_cancel+0x244/0x410 [ptlrpc] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.243136] [] ldlm_handle_enqueue0+0x65/0x10b0 [ptlrpc] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.263551] [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.292607] [] ldlm_handle_enqueue+0x66/0x70 [ptlrpc] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.313301] [] ? ldlm_server_completion_ast+0x0/0x6d0 [ptlrpc] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.342432] [] ? ost_blocking_ast+0x0/0x10f0 [ost] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.362846] [] ? ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.383594] [] ost_handle+0x1db8/0x48e0 [ost] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.412099] [] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.433200] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.462690] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.482846] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.503438] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.532284] [] ? __wake_up+0x53/0x70 Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.543604] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.572144] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.592530] [] child_rip+0xa/0x20 Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.603669] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.632358] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.652727] [] ? child_rip+0x0/0x20 Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.672338] Aug 21 15:33:25 atlas-oss1c7.ccs.ornl.gov kernel: [3800250.674008] LustreError: dumping log to /tmp/lustre-log.1408649605.15469 Aug 21 15:34:43 atlas-oss1c7.ccs.ornl.gov kernel: [3800328.273652] LustreError: 33046:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600 Aug 21 15:34:43 atlas-oss1c7.ccs.ornl.gov kernel: [3800328.273654] ns: filter-atlas1-OST0136_UUID lock: ffff880e19c69b40/0xecd0a12120f1c766 lrc: 4/0,0 mode: PW/PW res: [0x5af02a:0x0:0x0].0 rrc: 171 type: EXT [8388608->9175039] (req 8388608->8404991) flags: 0x20 nid: 13086@gni101 remote: 0xd8010812b1f71fab expref: 9 pid: 17904 timeout: 8093932058 lvb_type: 0 Aug 21 15:34:43 atlas-oss1c7.ccs.ornl.gov kernel: [3800328.403077] LustreError: 33046:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) Skipped 43 previous similar messages Aug 21 15:35:36 atlas-oss1c7.ccs.ornl.gov kernel: [3800381.076038] Lustre: atlas1-OST01c6: haven't heard from client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) in 1660 seconds. I think it's dead, and I am evicting it. exp ffff88034f196c00, cur 1408649736 expire 1408648836 last 1408648076 Aug 21 15:35:38 atlas-oss1c7.ccs.ornl.gov kernel: [3800383.076247] Lustre: atlas1-OST0136: haven't heard from client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) in 1662 seconds. I think it's dead, and I am evicting it. exp ffff88100be73c00, cur 1408649738 expire 1408648838 last 1408648076 Aug 21 15:35:39 atlas-oss1c7.ccs.ornl.gov kernel: [3800384.080184] Lustre: atlas1-OST00a6: haven't heard from client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) in 1663 seconds. I think it's dead, and I am evicting it. exp ffff880929febc00, cur 1408649739 expire 1408648839 last 1408648076 Aug 21 15:35:39 atlas-oss1c7.ccs.ornl.gov kernel: [3800384.144287] Lustre: Skipped 1 previous similar message Aug 21 15:35:43 atlas-oss1c7.ccs.ornl.gov kernel: [3800388.078229] Lustre: atlas1-OST0256: haven't heard from client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) in 1667 seconds. I think it's dead, and I am evicting it. exp ffff880faac4a800, cur 1408649743 expire 1408648843 last 1408648076