Jun 2 03:20:01 c210 systemd: Started Session 288 of user root. Jun 2 03:20:01 c210 systemd: Starting Session 288 of user root. Jun 2 03:30:01 c210 systemd: Started Session 289 of user root. Jun 2 03:30:01 c210 systemd: Starting Session 289 of user root. Jun 2 03:40:01 c210 systemd: Started Session 290 of user root. Jun 2 03:40:01 c210 systemd: Starting Session 290 of user root. Jun 2 03:50:01 c210 systemd: Started Session 291 of user root. Jun 2 03:50:01 c210 systemd: Starting Session 291 of user root. Jun 2 04:00:01 c210 systemd: Started Session 292 of user root. Jun 2 04:00:01 c210 systemd: Starting Session 292 of user root. Jun 2 04:01:01 c210 systemd: Started Session 293 of user root. Jun 2 04:01:01 c210 systemd: Starting Session 293 of user root. Jun 2 04:10:01 c210 systemd: Started Session 294 of user root. Jun 2 04:10:01 c210 systemd: Starting Session 294 of user root. Jun 2 04:20:01 c210 systemd: Started Session 295 of user root. Jun 2 04:20:01 c210 systemd: Starting Session 295 of user root. Jun 2 04:30:01 c210 systemd: Started Session 296 of user root. Jun 2 04:30:01 c210 systemd: Starting Session 296 of user root. Jun 2 04:40:01 c210 systemd: Started Session 297 of user root. Jun 2 04:40:01 c210 systemd: Starting Session 297 of user root. Jun 2 04:47:43 c210 systemd-logind: New session 298 of user root. Jun 2 04:47:43 c210 systemd: Started Session 298 of user root. Jun 2 04:47:43 c210 systemd: Starting Session 298 of user root. Jun 2 04:47:44 c210 systemd-logind: Removed session 298. Jun 2 04:48:43 c210 systemd-logind: New session 299 of user root. Jun 2 04:48:43 c210 systemd: Started Session 299 of user root. Jun 2 04:48:43 c210 systemd: Starting Session 299 of user root. Jun 2 04:50:01 c210 systemd: Started Session 300 of user root. Jun 2 04:50:01 c210 systemd: Starting Session 300 of user root. Jun 2 04:54:07 c210 systemd-logind: Removed session 299. Jun 2 04:58:01 c210 systemd-logind: New session 301 of user root. Jun 2 04:58:01 c210 systemd: Started Session 301 of user root. Jun 2 04:58:01 c210 systemd: Starting Session 301 of user root. Jun 2 05:00:01 c210 systemd: Started Session 302 of user root. Jun 2 05:00:01 c210 systemd: Starting Session 302 of user root. Jun 2 05:01:01 c210 systemd: Started Session 303 of user root. Jun 2 05:01:01 c210 systemd: Starting Session 303 of user root. Jun 2 05:01:14 c210 systemd-logind: Removed session 301. Jun 2 05:10:01 c210 systemd: Started Session 304 of user root. Jun 2 05:10:01 c210 systemd: Starting Session 304 of user root. Jun 2 05:20:01 c210 systemd: Started Session 305 of user root. Jun 2 05:20:01 c210 systemd: Starting Session 305 of user root. Jun 2 05:24:40 c210 systemd-logind: New session 306 of user root. Jun 2 05:24:40 c210 systemd: Started Session 306 of user root. Jun 2 05:24:40 c210 systemd: Starting Session 306 of user root. Jun 2 05:24:42 c210 systemd-logind: Removed session 306. Jun 2 05:25:02 c210 systemd-logind: New session 307 of user root. Jun 2 05:25:02 c210 systemd: Started Session 307 of user root. Jun 2 05:25:02 c210 systemd: Starting Session 307 of user root. Jun 2 05:30:01 c210 systemd: Started Session 308 of user root. Jun 2 05:30:01 c210 systemd: Starting Session 308 of user root. Jun 2 05:30:32 c210 systemd-logind: Removed session 307. Jun 2 05:40:01 c210 systemd: Started Session 309 of user root. Jun 2 05:40:01 c210 systemd: Starting Session 309 of user root. Jun 2 05:49:06 c210 kernel: Lustre: es90-MDT0002-mdc-ffffa04c3bdac000: Connection to es90-MDT0002 (at 172.16.254.192@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jun 2 05:49:06 c210 kernel: Lustre: Skipped 6 previous similar messages Jun 2 05:49:12 c210 kernel: Lustre: 25167:0:(client.c:2141:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1559476146/real 1559476146] req@ffffa04a56ba6300 x1635154568903840/t0(0) o9->es90-OST0000-osc-ffffa04c3bdac000@172.16.254.190@o2ib:28/4 lens 224/224 e 0 to 1 dl 1559476152 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 2 05:49:12 c210 kernel: Lustre: 25167:0:(client.c:2141:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jun 2 05:49:38 c210 kernel: LustreError: 166-1: MGC172.16.254.190@o2ib: Connection to MGS (at 172.16.254.190@o2ib) was lost; in progress operations using this service will fail Jun 2 05:49:40 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 172.16.254.190@o2ib: 3 seconds Jun 2 05:49:40 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 8 previous similar messages Jun 2 05:50:01 c210 systemd: Started Session 310 of user root. Jun 2 05:50:01 c210 systemd: Starting Session 310 of user root. Jun 2 05:50:29 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 172.16.254.190@o2ib: 77 seconds Jun 2 05:50:29 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 9 previous similar messages Jun 2 05:51:44 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 172.16.254.160@o2ib: 152 seconds Jun 2 05:51:44 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 26 previous similar messages Jun 2 05:54:14 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 172.16.254.190@o2ib: 302 seconds Jun 2 05:54:14 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 53 previous similar messages Jun 2 05:58:50 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 172.16.254.160@o2ib: 553 seconds Jun 2 05:58:50 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 98 previous similar messages Jun 2 06:00:01 c210 systemd: Started Session 311 of user root. Jun 2 06:00:01 c210 systemd: Starting Session 311 of user root. Jun 2 06:01:01 c210 systemd: Started Session 312 of user root. Jun 2 06:01:01 c210 systemd: Starting Session 312 of user root. Jun 2 06:05:56 c210 kernel: Lustre: Evicted from MGS (at MGC172.16.254.190@o2ib_0) after server handle changed from 0x47825270126d2367 to 0xa30b812944e7a39e Jun 2 06:05:56 c210 kernel: Lustre: MGC172.16.254.190@o2ib: Connection restored to MGC172.16.254.190@o2ib_0 (at 172.16.254.190@o2ib) Jun 2 06:07:13 c210 kernel: Lustre: 25167:0:(client.c:2141:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1559477176/real 1559477176] req@ffffa042d134e780 x1635154568910432/t0(0) o400->es90-MDT0003-mdc-ffffa04c3bdac000@172.16.254.193@o2ib:12/10 lens 224/224 e 2 to 1 dl 1559477233 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1 Jun 2 06:07:13 c210 kernel: Lustre: 25167:0:(client.c:2141:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jun 2 06:07:18 c210 kernel: Lustre: 25179:0:(client.c:2141:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1559477231/real 1559477231] req@ffffa041aeb7ad00 x1635154568910592/t0(0) o400->MGC172.16.254.190@o2ib@172.16.254.190@o2ib:26/25 lens 224/224 e 0 to 1 dl 1559477238 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 2 06:07:18 c210 kernel: Lustre: 25179:0:(client.c:2141:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jun 2 06:07:18 c210 kernel: LustreError: 166-1: MGC172.16.254.190@o2ib: Connection to MGS (at 172.16.254.190@o2ib) was lost; in progress operations using this service will fail Jun 2 06:07:24 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 172.16.254.163@o2ib: 5 seconds Jun 2 06:07:24 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 155 previous similar messages Jun 2 06:07:30 c210 kernel: Lustre: 25167:0:(client.c:2141:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1559477168/real 1559477168] req@ffffa042d134de80 x1635154568910240/t0(0) o400->es90-MDT0006-mdc-ffffa04c3bdac000@172.16.254.162@o2ib:12/10 lens 224/224 e 3 to 1 dl 1559477250 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1 Jun 2 06:07:30 c210 kernel: Lustre: 25167:0:(client.c:2141:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jun 2 06:10:01 c210 systemd: Started Session 313 of user root. Jun 2 06:10:01 c210 systemd: Starting Session 313 of user root. Jun 2 06:17:46 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 172.16.254.162@o2ib: 627 seconds Jun 2 06:17:46 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 214 previous similar messages Jun 2 06:20:01 c210 systemd: Started Session 314 of user root. Jun 2 06:20:01 c210 systemd: Starting Session 314 of user root. Jun 2 06:27:47 c210 kernel: Lustre: Evicted from MGS (at MGC172.16.254.190@o2ib_0) after server handle changed from 0xa30b812944e7a39e to 0x57b6ec29bda58dc Jun 2 06:27:47 c210 kernel: Lustre: MGC172.16.254.190@o2ib: Connection restored to MGC172.16.254.190@o2ib_0 (at 172.16.254.190@o2ib) Jun 2 06:28:35 c210 kernel: Lustre: es90-MDT0006-mdc-ffffa04c3bdac000: Connection restored to 172.16.254.162@o2ib (at 172.16.254.162@o2ib) Jun 2 06:29:20 c210 systemd-logind: New session 315 of user root. Jun 2 06:29:20 c210 systemd: Started Session 315 of user root. Jun 2 06:29:20 c210 systemd: Starting Session 315 of user root. Jun 2 06:29:21 c210 kernel: Lustre: Unmounted es90-client Jun 2 06:29:21 c210 systemd-logind: Removed session 315. Jun 2 06:29:33 c210 systemd-logind: New session 316 of user root. Jun 2 06:29:33 c210 systemd: Started Session 316 of user root. Jun 2 06:29:33 c210 systemd: Starting Session 316 of user root. Jun 2 06:29:34 c210 kernel: LustreError: 11-0: es90-OST0000-osc-ffffa04c3d0ed000: operation ost_connect to node 172.16.254.190@o2ib failed: rc = -16 Jun 2 06:29:34 c210 kernel: Lustre: Mounted es90-client Jun 2 06:29:34 c210 systemd-logind: Removed session 316. Jun 2 06:29:59 c210 kernel: LustreError: 11-0: es90-OST0003-osc-ffffa04c3d0ed000: operation ost_connect to node 172.16.254.191@o2ib failed: rc = -16 Jun 2 06:29:59 c210 kernel: LustreError: Skipped 16 previous similar messages Jun 2 06:30:01 c210 systemd: Started Session 317 of user root. Jun 2 06:30:01 c210 systemd: Starting Session 317 of user root. Jun 2 06:30:24 c210 kernel: LustreError: 11-0: es90-OST0003-osc-ffffa04c3d0ed000: operation ost_connect to node 172.16.254.191@o2ib failed: rc = -16 Jun 2 06:30:24 c210 kernel: LustreError: Skipped 14 previous similar messages Jun 2 06:30:49 c210 kernel: LustreError: 11-0: es90-OST0005-osc-ffffa04c3d0ed000: operation ost_connect to node 172.16.254.192@o2ib failed: rc = -16 Jun 2 06:30:49 c210 kernel: LustreError: Skipped 14 previous similar messages Jun 2 06:31:14 c210 kernel: LustreError: 11-0: es90-OST0006-osc-ffffa04c3d0ed000: operation ost_connect to node 172.16.254.193@o2ib failed: rc = -16 Jun 2 06:31:14 c210 kernel: LustreError: Skipped 14 previous similar messages Jun 2 06:32:04 c210 kernel: LustreError: 11-0: es90-OST0002-osc-ffffa04c3d0ed000: operation ost_connect to node 172.16.254.191@o2ib failed: rc = -16 Jun 2 06:32:04 c210 kernel: LustreError: Skipped 29 previous similar messages Jun 2 06:33:46 c210 systemd-logind: New session 318 of user root. Jun 2 06:33:46 c210 systemd: Started Session 318 of user root. Jun 2 06:33:46 c210 systemd: Starting Session 318 of user root. Jun 2 06:33:46 c210 kernel: Lustre: Unmounted es90-client Jun 2 06:33:46 c210 systemd-logind: Removed session 318. Jun 2 06:33:46 c210 systemd-logind: New session 319 of user root. Jun 2 06:33:46 c210 systemd: Started Session 319 of user root. Jun 2 06:33:46 c210 systemd: Starting Session 319 of user root. Jun 2 06:33:46 c210 kernel: Lustre: Mounted es90-client Jun 2 06:33:46 c210 systemd-logind: Removed session 319. Jun 2 06:33:49 c210 systemd-logind: New session 320 of user root. Jun 2 06:33:49 c210 systemd: Started Session 320 of user root. Jun 2 06:33:49 c210 systemd: Starting Session 320 of user root. Jun 2 06:33:49 c210 systemd-logind: Removed session 320. Jun 2 06:33:51 c210 systemd-logind: New session 321 of user root. Jun 2 06:33:51 c210 systemd: Started Session 321 of user root. Jun 2 06:33:51 c210 systemd: Starting Session 321 of user root. Jun 2 06:33:51 c210 systemd-logind: Removed session 321. Jun 2 06:35:22 c210 systemd-logind: New session 322 of user root. Jun 2 06:35:22 c210 systemd: Started Session 322 of user root. Jun 2 06:35:22 c210 systemd: Starting Session 322 of user root. Jun 2 06:35:22 c210 systemd-logind: Removed session 322. Jun 2 06:36:12 c210 systemd-logind: New session 323 of user root. Jun 2 06:36:12 c210 systemd: Started Session 323 of user root. Jun 2 06:36:12 c210 systemd: Starting Session 323 of user root. Jun 2 06:36:12 c210 kernel: Lustre: Unmounted es90-client Jun 2 06:36:12 c210 systemd-logind: Removed session 323. Jun 2 06:36:12 c210 systemd-logind: New session 324 of user root. Jun 2 06:36:12 c210 systemd: Started Session 324 of user root. Jun 2 06:36:12 c210 systemd: Starting Session 324 of user root. Jun 2 06:36:12 c210 kernel: Lustre: Mounted es90-client Jun 2 06:36:12 c210 systemd-logind: Removed session 324. Jun 2 06:36:14 c210 systemd-logind: New session 325 of user root. Jun 2 06:36:14 c210 systemd: Started Session 325 of user root. Jun 2 06:36:14 c210 systemd: Starting Session 325 of user root. Jun 2 06:36:14 c210 systemd-logind: Removed session 325. Jun 2 06:36:17 c210 systemd-logind: New session 326 of user root. Jun 2 06:36:17 c210 systemd: Started Session 326 of user root. Jun 2 06:36:17 c210 systemd: Starting Session 326 of user root. Jun 2 06:36:17 c210 systemd-logind: Removed session 326. Jun 2 06:40:01 c210 systemd: Started Session 327 of user root. Jun 2 06:40:01 c210 systemd: Starting Session 327 of user root. Jun 2 06:50:01 c210 systemd: Started Session 328 of user root. Jun 2 06:50:01 c210 systemd: Starting Session 328 of user root. Jun 2 06:52:08 c210 systemd-logind: New session 329 of user root. Jun 2 06:52:08 c210 systemd: Started Session 329 of user root. Jun 2 06:52:08 c210 systemd: Starting Session 329 of user root. Jun 2 06:52:08 c210 systemd-logind: Removed session 329. Jun 2 06:52:30 c210 systemd-logind: New session 330 of user root. Jun 2 06:52:30 c210 systemd: Started Session 330 of user root. Jun 2 06:52:30 c210 systemd: Starting Session 330 of user root. Jun 2 06:52:30 c210 kernel: Lustre: Unmounted es90-client Jun 2 06:52:30 c210 systemd-logind: Removed session 330. Jun 2 06:52:30 c210 systemd-logind: New session 331 of user root. Jun 2 06:52:30 c210 systemd: Started Session 331 of user root. Jun 2 06:52:30 c210 systemd: Starting Session 331 of user root. Jun 2 06:52:31 c210 kernel: Lustre: Mounted es90-client Jun 2 06:52:31 c210 systemd-logind: Removed session 331. Jun 2 06:52:33 c210 systemd-logind: New session 332 of user root. Jun 2 06:52:33 c210 systemd: Started Session 332 of user root. Jun 2 06:52:33 c210 systemd: Starting Session 332 of user root. Jun 2 06:52:33 c210 systemd-logind: Removed session 332. Jun 2 06:52:35 c210 systemd-logind: New session 333 of user root. Jun 2 06:52:35 c210 systemd: Started Session 333 of user root. Jun 2 06:52:35 c210 systemd: Starting Session 333 of user root. Jun 2 06:52:35 c210 systemd-logind: Removed session 333. Jun 2 06:52:35 c210 systemd-logind: New session 334 of user root. Jun 2 06:52:35 c210 systemd: Started Session 334 of user root. Jun 2 06:52:35 c210 systemd: Starting Session 334 of user root. Jun 2 06:52:35 c210 systemd-logind: Removed session 334. Jun 2 06:52:43 c210 systemd-logind: New session 335 of user root. Jun 2 06:52:43 c210 systemd: Started Session 335 of user root. Jun 2 06:52:43 c210 systemd: Starting Session 335 of user root. Jun 2 06:52:44 c210 kernel: Lustre: Unmounted es90-client Jun 2 06:52:44 c210 systemd-logind: Removed session 335. Jun 2 06:52:44 c210 systemd-logind: New session 336 of user root. Jun 2 06:52:44 c210 systemd: Started Session 336 of user root. Jun 2 06:52:44 c210 systemd: Starting Session 336 of user root. Jun 2 06:52:44 c210 kernel: Lustre: Mounted es90-client Jun 2 06:52:44 c210 systemd-logind: Removed session 336. Jun 2 06:52:46 c210 systemd-logind: New session 337 of user root. Jun 2 06:52:46 c210 systemd: Started Session 337 of user root. Jun 2 06:52:46 c210 systemd: Starting Session 337 of user root. Jun 2 06:52:46 c210 systemd-logind: Removed session 337. Jun 2 06:52:48 c210 systemd-logind: New session 338 of user root. Jun 2 06:52:48 c210 systemd: Started Session 338 of user root. Jun 2 06:52:48 c210 systemd: Starting Session 338 of user root. Jun 2 06:52:48 c210 systemd-logind: Removed session 338. Jun 2 06:52:49 c210 systemd-logind: New session 339 of user root. Jun 2 06:52:49 c210 systemd: Started Session 339 of user root. Jun 2 06:52:49 c210 systemd: Starting Session 339 of user root. Jun 2 06:52:49 c210 systemd-logind: Removed session 339. Jun 2 06:52:49 c210 systemd-logind: New session 340 of user root. Jun 2 06:52:49 c210 systemd: Started Session 340 of user root. Jun 2 06:52:49 c210 systemd: Starting Session 340 of user root. Jun 2 06:57:55 c210 systemd-logind: Removed session 340. Jun 2 06:57:55 c210 systemd-logind: New session 341 of user root. Jun 2 06:57:55 c210 systemd: Started Session 341 of user root. Jun 2 06:57:55 c210 systemd: Starting Session 341 of user root. Jun 2 07:00:01 c210 systemd: Started Session 342 of user root. Jun 2 07:00:01 c210 systemd: Starting Session 342 of user root. Jun 2 07:01:01 c210 systemd: Started Session 343 of user root. Jun 2 07:01:01 c210 systemd: Starting Session 343 of user root. Jun 2 07:03:20 c210 systemd-logind: Removed session 341. Jun 2 07:03:20 c210 systemd-logind: New session 344 of user root. Jun 2 07:03:20 c210 systemd: Started Session 344 of user root. Jun 2 07:03:20 c210 systemd: Starting Session 344 of user root. Jun 2 07:03:56 c210 systemd-logind: Removed session 344. Jun 2 07:03:56 c210 systemd-logind: New session 345 of user root. Jun 2 07:03:56 c210 systemd: Started Session 345 of user root. Jun 2 07:03:56 c210 systemd: Starting Session 345 of user root. Jun 2 07:04:23 c210 systemd-logind: Removed session 345. Jun 2 07:04:23 c210 systemd-logind: New session 346 of user root. Jun 2 07:04:23 c210 systemd: Started Session 346 of user root. Jun 2 07:04:23 c210 systemd: Starting Session 346 of user root. Jun 2 07:04:34 c210 systemd-logind: Removed session 346. Jun 2 07:04:34 c210 systemd-logind: New session 347 of user root. Jun 2 07:04:34 c210 systemd: Started Session 347 of user root. Jun 2 07:04:34 c210 systemd: Starting Session 347 of user root. Jun 2 07:10:01 c210 systemd: Started Session 348 of user root. Jun 2 07:10:01 c210 systemd: Starting Session 348 of user root. Jun 2 07:11:26 c210 systemd-logind: Removed session 347. Jun 2 07:11:26 c210 systemd-logind: New session 349 of user root. Jun 2 07:11:26 c210 systemd: Started Session 349 of user root. Jun 2 07:11:26 c210 systemd: Starting Session 349 of user root. Jun 2 07:11:55 c210 systemd-logind: Removed session 349. Jun 2 07:11:55 c210 systemd-logind: New session 350 of user root. Jun 2 07:11:55 c210 systemd: Started Session 350 of user root. Jun 2 07:11:55 c210 systemd: Starting Session 350 of user root. Jun 2 07:18:47 c210 systemd-logind: Removed session 350. Jun 2 07:20:01 c210 systemd: Started Session 351 of user root. Jun 2 07:20:01 c210 systemd: Starting Session 351 of user root. Jun 2 07:20:10 c210 systemd-logind: New session 352 of user root. Jun 2 07:20:10 c210 systemd: Started Session 352 of user root. Jun 2 07:20:10 c210 systemd: Starting Session 352 of user root. Jun 2 07:20:13 c210 kernel: Lustre: Unmounted es90-client Jun 2 07:20:13 c210 systemd-logind: Removed session 352. Jun 2 07:20:14 c210 systemd-logind: New session 353 of user root. Jun 2 07:20:14 c210 systemd: Started Session 353 of user root. Jun 2 07:20:14 c210 systemd: Starting Session 353 of user root. Jun 2 07:20:14 c210 kernel: Lustre: Mounted es90-client Jun 2 07:20:14 c210 systemd-logind: Removed session 353. Jun 2 07:20:16 c210 systemd-logind: New session 354 of user root. Jun 2 07:20:16 c210 systemd: Started Session 354 of user root. Jun 2 07:20:16 c210 systemd: Starting Session 354 of user root. Jun 2 07:20:16 c210 systemd-logind: Removed session 354. Jun 2 07:20:18 c210 systemd-logind: New session 355 of user root. Jun 2 07:20:18 c210 systemd: Started Session 355 of user root. Jun 2 07:20:18 c210 systemd: Starting Session 355 of user root. Jun 2 07:20:18 c210 systemd-logind: Removed session 355. Jun 2 07:20:22 c210 systemd-logind: New session 356 of user root. Jun 2 07:20:22 c210 systemd: Started Session 356 of user root. Jun 2 07:20:22 c210 systemd: Starting Session 356 of user root. Jun 2 07:20:22 c210 systemd-logind: Removed session 356. Jun 2 07:20:53 c210 systemd: Starting Cleanup of Temporary Directories... Jun 2 07:20:53 c210 systemd: Started Cleanup of Temporary Directories. Jun 2 07:26:37 c210 systemd-logind: New session 357 of user root. Jun 2 07:26:37 c210 systemd: Started Session 357 of user root. Jun 2 07:26:37 c210 systemd: Starting Session 357 of user root. Jun 2 07:26:38 c210 kernel: Lustre: Unmounted es90-client Jun 2 07:26:38 c210 systemd-logind: Removed session 357. Jun 2 07:26:38 c210 systemd-logind: New session 358 of user root. Jun 2 07:26:38 c210 systemd: Started Session 358 of user root. Jun 2 07:26:38 c210 systemd: Starting Session 358 of user root. Jun 2 07:26:38 c210 kernel: Lustre: Mounted es90-client Jun 2 07:26:38 c210 systemd-logind: Removed session 358. Jun 2 07:26:40 c210 systemd-logind: New session 359 of user root. Jun 2 07:26:40 c210 systemd: Started Session 359 of user root. Jun 2 07:26:40 c210 systemd: Starting Session 359 of user root. Jun 2 07:26:40 c210 systemd-logind: Removed session 359. Jun 2 07:26:42 c210 systemd-logind: New session 360 of user root. Jun 2 07:26:42 c210 systemd: Started Session 360 of user root. Jun 2 07:26:42 c210 systemd: Starting Session 360 of user root. Jun 2 07:26:42 c210 systemd-logind: Removed session 360. Jun 2 07:26:43 c210 systemd-logind: New session 361 of user root. Jun 2 07:26:43 c210 systemd: Started Session 361 of user root. Jun 2 07:26:43 c210 systemd: Starting Session 361 of user root. Jun 2 07:26:43 c210 systemd-logind: Removed session 361. Jun 2 07:26:43 c210 systemd-logind: New session 362 of user root. Jun 2 07:26:43 c210 systemd: Started Session 362 of user root. Jun 2 07:26:43 c210 systemd: Starting Session 362 of user root. Jun 2 07:30:01 c210 systemd: Started Session 363 of user root. Jun 2 07:30:01 c210 systemd: Starting Session 363 of user root. Jun 2 07:32:04 c210 systemd-logind: Removed session 362. Jun 2 07:32:04 c210 systemd-logind: New session 364 of user root. Jun 2 07:32:04 c210 systemd: Started Session 364 of user root. Jun 2 07:32:04 c210 systemd: Starting Session 364 of user root. Jun 2 07:37:30 c210 systemd-logind: Removed session 364. Jun 2 07:37:30 c210 systemd-logind: New session 365 of user root. Jun 2 07:37:30 c210 systemd: Started Session 365 of user root. Jun 2 07:37:30 c210 systemd: Starting Session 365 of user root. Jun 2 07:37:57 c210 systemd-logind: Removed session 365. Jun 2 07:37:57 c210 systemd-logind: New session 366 of user root. Jun 2 07:37:57 c210 systemd: Started Session 366 of user root. Jun 2 07:37:57 c210 systemd: Starting Session 366 of user root. Jun 2 07:38:02 c210 systemd-logind: Removed session 366. Jun 2 07:38:02 c210 systemd-logind: New session 367 of user root. Jun 2 07:38:02 c210 systemd: Started Session 367 of user root. Jun 2 07:38:02 c210 systemd: Starting Session 367 of user root. Jun 2 07:40:01 c210 systemd: Started Session 368 of user root. Jun 2 07:40:01 c210 systemd: Starting Session 368 of user root. Jun 2 07:44:55 c210 systemd-logind: Removed session 367. Jun 2 07:44:55 c210 systemd-logind: New session 369 of user root. Jun 2 07:44:55 c210 systemd: Started Session 369 of user root. Jun 2 07:44:55 c210 systemd: Starting Session 369 of user root. Jun 2 07:45:24 c210 systemd-logind: Removed session 369. Jun 2 07:45:24 c210 systemd-logind: New session 370 of user root. Jun 2 07:45:24 c210 systemd: Started Session 370 of user root. Jun 2 07:45:24 c210 systemd: Starting Session 370 of user root. Jun 2 07:50:01 c210 systemd: Started Session 371 of user root. Jun 2 07:50:01 c210 systemd: Starting Session 371 of user root. Jun 2 07:52:09 c210 systemd-logind: Removed session 370. Jun 2 07:57:36 c210 systemd-logind: New session 372 of user root. Jun 2 07:57:36 c210 systemd: Started Session 372 of user root. Jun 2 07:57:36 c210 systemd: Starting Session 372 of user root. Jun 2 07:57:36 c210 kernel: Lustre: Unmounted es90-client Jun 2 07:57:36 c210 systemd-logind: Removed session 372. Jun 2 07:57:36 c210 systemd-logind: New session 373 of user root. Jun 2 07:57:36 c210 systemd: Started Session 373 of user root. Jun 2 07:57:36 c210 systemd: Starting Session 373 of user root. Jun 2 07:57:36 c210 kernel: Lustre: Mounted es90-client Jun 2 07:57:36 c210 systemd-logind: Removed session 373. Jun 2 07:57:38 c210 systemd-logind: New session 374 of user root. Jun 2 07:57:38 c210 systemd: Started Session 374 of user root. Jun 2 07:57:38 c210 systemd: Starting Session 374 of user root. Jun 2 07:57:38 c210 systemd-logind: Removed session 374. Jun 2 07:57:40 c210 systemd-logind: New session 375 of user root. Jun 2 07:57:40 c210 systemd: Started Session 375 of user root. Jun 2 07:57:40 c210 systemd: Starting Session 375 of user root. Jun 2 07:57:40 c210 systemd-logind: Removed session 375. Jun 2 07:57:44 c210 systemd-logind: New session 376 of user root. Jun 2 07:57:44 c210 systemd: Started Session 376 of user root. Jun 2 07:57:44 c210 systemd: Starting Session 376 of user root. Jun 2 07:57:44 c210 systemd-logind: Removed session 376. Jun 2 07:57:44 c210 systemd-logind: New session 377 of user root. Jun 2 07:57:44 c210 systemd: Started Session 377 of user root. Jun 2 07:57:44 c210 systemd: Starting Session 377 of user root. Jun 2 08:00:01 c210 systemd: Started Session 378 of user root. Jun 2 08:00:01 c210 systemd: Starting Session 378 of user root. Jun 2 08:01:01 c210 systemd: Started Session 379 of user root. Jun 2 08:01:01 c210 systemd: Starting Session 379 of user root. Jun 2 08:03:04 c210 systemd-logind: Removed session 377. Jun 2 08:03:04 c210 systemd-logind: New session 380 of user root. Jun 2 08:03:04 c210 systemd: Started Session 380 of user root. Jun 2 08:03:04 c210 systemd: Starting Session 380 of user root. Jun 2 08:08:31 c210 systemd-logind: Removed session 380. Jun 2 08:08:31 c210 systemd-logind: New session 381 of user root. Jun 2 08:08:31 c210 systemd: Started Session 381 of user root. Jun 2 08:08:31 c210 systemd: Starting Session 381 of user root. Jun 2 08:09:05 c210 systemd-logind: Removed session 381. Jun 2 08:09:05 c210 systemd-logind: New session 382 of user root. Jun 2 08:09:05 c210 systemd: Started Session 382 of user root. Jun 2 08:09:05 c210 systemd: Starting Session 382 of user root. Jun 2 08:09:33 c210 systemd-logind: Removed session 382. Jun 2 08:09:33 c210 systemd-logind: New session 383 of user root. Jun 2 08:09:33 c210 systemd: Started Session 383 of user root. Jun 2 08:09:33 c210 systemd: Starting Session 383 of user root. Jun 2 08:09:44 c210 systemd-logind: Removed session 383. Jun 2 08:09:44 c210 systemd-logind: New session 384 of user root. Jun 2 08:09:44 c210 systemd: Started Session 384 of user root. Jun 2 08:09:44 c210 systemd: Starting Session 384 of user root. Jun 2 08:10:01 c210 systemd: Started Session 385 of user root. Jun 2 08:10:01 c210 systemd: Starting Session 385 of user root. Jun 2 08:16:47 c210 systemd-logind: Removed session 384. Jun 2 08:16:48 c210 systemd-logind: New session 386 of user root. Jun 2 08:16:48 c210 systemd: Started Session 386 of user root. Jun 2 08:16:48 c210 systemd: Starting Session 386 of user root. Jun 2 08:17:16 c210 systemd-logind: Removed session 386. Jun 2 08:17:16 c210 systemd-logind: New session 387 of user root. Jun 2 08:17:16 c210 systemd: Started Session 387 of user root. Jun 2 08:17:16 c210 systemd: Starting Session 387 of user root. Jun 2 08:20:01 c210 systemd: Started Session 388 of user root. Jun 2 08:20:01 c210 systemd: Starting Session 388 of user root. Jun 2 08:24:06 c210 systemd-logind: Removed session 387. Jun 2 08:30:01 c210 systemd: Started Session 389 of user root. Jun 2 08:30:01 c210 systemd: Starting Session 389 of user root. Jun 2 08:40:01 c210 systemd: Started Session 390 of user root. Jun 2 08:40:01 c210 systemd: Starting Session 390 of user root. Jun 2 08:50:01 c210 systemd: Started Session 391 of user root. Jun 2 08:50:01 c210 systemd: Starting Session 391 of user root. Jun 2 09:00:01 c210 systemd: Started Session 392 of user root. Jun 2 09:00:01 c210 systemd: Starting Session 392 of user root. Jun 2 09:01:01 c210 systemd: Started Session 393 of user root. Jun 2 09:01:01 c210 systemd: Starting Session 393 of user root. Jun 2 09:10:01 c210 systemd: Started Session 394 of user root. Jun 2 09:10:01 c210 systemd: Starting Session 394 of user root. Jun 2 09:20:01 c210 systemd: Started Session 395 of user root. Jun 2 09:20:01 c210 systemd: Starting Session 395 of user root. Jun 2 09:27:11 c210 systemd-logind: New session 396 of user root. Jun 2 09:27:11 c210 systemd: Started Session 396 of user root. Jun 2 09:27:11 c210 systemd: Starting Session 396 of user root. Jun 2 09:27:14 c210 kernel: Lustre: Unmounted es90-client Jun 2 09:27:14 c210 systemd-logind: Removed session 396. Jun 2 09:28:03 c210 systemd-logind: New session 397 of user root. Jun 2 09:28:03 c210 systemd: Started Session 397 of user root. Jun 2 09:28:03 c210 systemd: Starting Session 397 of user root. Jun 2 09:28:03 c210 kernel: Lustre: Mounted es90-client Jun 2 09:28:03 c210 systemd-logind: Removed session 397. Jun 2 09:28:06 c210 systemd-logind: New session 398 of user root. Jun 2 09:28:06 c210 systemd: Started Session 398 of user root. Jun 2 09:28:06 c210 systemd: Starting Session 398 of user root. Jun 2 09:28:06 c210 systemd-logind: Removed session 398. Jun 2 09:28:08 c210 systemd-logind: New session 399 of user root. Jun 2 09:28:08 c210 systemd: Started Session 399 of user root. Jun 2 09:28:08 c210 systemd: Starting Session 399 of user root. Jun 2 09:28:08 c210 systemd-logind: Removed session 399. Jun 2 09:30:01 c210 systemd: Started Session 400 of user root. Jun 2 09:30:01 c210 systemd: Starting Session 400 of user root. Jun 2 09:40:01 c210 systemd: Started Session 401 of user root. Jun 2 09:40:01 c210 systemd: Starting Session 401 of user root. Jun 2 09:50:01 c210 systemd: Started Session 402 of user root. Jun 2 09:50:01 c210 systemd: Starting Session 402 of user root. Jun 2 10:00:01 c210 systemd: Started Session 403 of user root. Jun 2 10:00:01 c210 systemd: Starting Session 403 of user root. Jun 2 10:01:01 c210 systemd: Started Session 404 of user root. Jun 2 10:01:01 c210 systemd: Starting Session 404 of user root. Jun 2 10:10:01 c210 systemd: Started Session 405 of user root. Jun 2 10:10:01 c210 systemd: Starting Session 405 of user root. Jun 2 10:20:01 c210 systemd: Started Session 406 of user root. Jun 2 10:20:01 c210 systemd: Starting Session 406 of user root. Jun 2 10:28:31 c210 systemd-logind: New session 407 of user root. Jun 2 10:28:31 c210 systemd: Started Session 407 of user root. Jun 2 10:28:31 c210 systemd: Starting Session 407 of user root. Jun 2 10:28:31 c210 systemd-logind: Removed session 407. Jun 2 10:28:32 c210 systemd-logind: New session 408 of user root. Jun 2 10:28:32 c210 systemd: Started Session 408 of user root. Jun 2 10:28:32 c210 systemd: Starting Session 408 of user root. Jun 2 10:30:01 c210 systemd: Started Session 409 of user root. Jun 2 10:30:01 c210 systemd: Starting Session 409 of user root. Jun 2 10:34:05 c210 systemd-logind: Removed session 408. Jun 2 10:34:05 c210 systemd-logind: New session 410 of user root. Jun 2 10:34:05 c210 systemd: Started Session 410 of user root. Jun 2 10:34:05 c210 systemd: Starting Session 410 of user root. Jun 2 10:39:12 c210 systemd-logind: Removed session 410. Jun 2 10:39:12 c210 systemd-logind: New session 411 of user root. Jun 2 10:39:12 c210 systemd: Started Session 411 of user root. Jun 2 10:39:12 c210 systemd: Starting Session 411 of user root. Jun 2 10:40:01 c210 systemd: Started Session 412 of user root. Jun 2 10:40:01 c210 systemd: Starting Session 412 of user root. Jun 2 10:46:03 c210 systemd-logind: Removed session 411. Jun 2 10:46:03 c210 systemd-logind: New session 413 of user root. Jun 2 10:46:03 c210 systemd: Started Session 413 of user root. Jun 2 10:46:03 c210 systemd: Starting Session 413 of user root. Jun 2 10:50:01 c210 systemd: Started Session 414 of user root. Jun 2 10:50:01 c210 systemd: Starting Session 414 of user root. Jun 2 10:51:42 c210 systemd-logind: Removed session 413. Jun 2 10:51:43 c210 systemd-logind: New session 415 of user root. Jun 2 10:51:43 c210 systemd: Started Session 415 of user root. Jun 2 10:51:43 c210 systemd: Starting Session 415 of user root. Jun 2 10:52:17 c210 systemd-logind: Removed session 415. Jun 2 10:52:18 c210 systemd-logind: New session 416 of user root. Jun 2 10:52:18 c210 systemd: Started Session 416 of user root. Jun 2 10:52:18 c210 systemd: Starting Session 416 of user root. Jun 2 10:56:35 c210 systemd-logind: Removed session 416. Jun 2 10:56:35 c210 systemd-logind: New session 417 of user root. Jun 2 10:56:35 c210 systemd: Started Session 417 of user root. Jun 2 10:56:35 c210 systemd: Starting Session 417 of user root. Jun 2 10:57:06 c210 systemd-logind: Removed session 417. Jun 2 10:57:06 c210 systemd-logind: New session 418 of user root. Jun 2 10:57:06 c210 systemd: Started Session 418 of user root. Jun 2 10:57:06 c210 systemd: Starting Session 418 of user root. Jun 2 11:00:01 c210 systemd: Started Session 419 of user root. Jun 2 11:00:01 c210 systemd: Starting Session 419 of user root. Jun 2 11:01:01 c210 systemd: Started Session 420 of user root. Jun 2 11:01:01 c210 systemd: Starting Session 420 of user root. Jun 2 11:01:10 c210 systemd-logind: Removed session 418. Jun 2 11:01:10 c210 systemd-logind: New session 421 of user root. Jun 2 11:01:10 c210 systemd: Started Session 421 of user root. Jun 2 11:01:10 c210 systemd: Starting Session 421 of user root. Jun 2 11:01:21 c210 systemd-logind: Removed session 421. Jun 2 11:01:21 c210 systemd-logind: New session 422 of user root. Jun 2 11:01:21 c210 systemd: Started Session 422 of user root. Jun 2 11:01:21 c210 systemd: Starting Session 422 of user root. Jun 2 11:08:50 c210 systemd-logind: Removed session 422. Jun 2 11:08:50 c210 systemd-logind: New session 423 of user root. Jun 2 11:08:50 c210 systemd: Started Session 423 of user root. Jun 2 11:08:50 c210 systemd: Starting Session 423 of user root. Jun 2 11:09:20 c210 systemd-logind: Removed session 423. Jun 2 11:09:20 c210 systemd-logind: New session 424 of user root. Jun 2 11:09:20 c210 systemd: Started Session 424 of user root. Jun 2 11:09:20 c210 systemd: Starting Session 424 of user root. Jun 2 11:10:01 c210 systemd: Started Session 425 of user root. Jun 2 11:10:01 c210 systemd: Starting Session 425 of user root. Jun 2 11:16:54 c210 systemd-logind: Removed session 424. Jun 2 11:20:01 c210 systemd: Started Session 426 of user root. Jun 2 11:20:01 c210 systemd: Starting Session 426 of user root. Jun 2 11:30:01 c210 systemd: Started Session 427 of user root. Jun 2 11:30:01 c210 systemd: Starting Session 427 of user root. Jun 2 11:40:01 c210 systemd: Started Session 428 of user root. Jun 2 11:40:01 c210 systemd: Starting Session 428 of user root. Jun 2 11:50:01 c210 systemd: Started Session 429 of user root. Jun 2 11:50:01 c210 systemd: Starting Session 429 of user root. Jun 2 12:00:01 c210 systemd: Started Session 430 of user root. Jun 2 12:00:01 c210 systemd: Starting Session 430 of user root. Jun 2 12:01:01 c210 systemd: Started Session 431 of user root. Jun 2 12:01:01 c210 systemd: Starting Session 431 of user root. Jun 2 12:10:01 c210 systemd: Started Session 432 of user root. Jun 2 12:10:01 c210 systemd: Starting Session 432 of user root. Jun 2 12:20:01 c210 systemd: Started Session 433 of user root. Jun 2 12:20:01 c210 systemd: Starting Session 433 of user root. Jun 2 12:30:01 c210 systemd: Started Session 434 of user root. Jun 2 12:30:01 c210 systemd: Starting Session 434 of user root. Jun 2 12:33:18 c210 chronyd[960]: Selected source 216.229.4.66 Jun 2 12:40:01 c210 systemd: Started Session 435 of user root. Jun 2 12:40:01 c210 systemd: Starting Session 435 of user root. Jun 2 12:50:01 c210 systemd: Started Session 436 of user root. Jun 2 12:50:01 c210 systemd: Starting Session 436 of user root. Jun 2 13:00:01 c210 systemd: Started Session 437 of user root. Jun 2 13:00:01 c210 systemd: Starting Session 437 of user root. Jun 2 13:01:01 c210 systemd: Started Session 438 of user root. Jun 2 13:01:01 c210 systemd: Starting Session 438 of user root. Jun 2 13:10:01 c210 systemd: Started Session 439 of user root. Jun 2 13:10:01 c210 systemd: Starting Session 439 of user root. Jun 2 13:20:01 c210 systemd: Started Session 440 of user root. Jun 2 13:20:01 c210 systemd: Starting Session 440 of user root. Jun 2 13:30:01 c210 systemd: Started Session 441 of user root. Jun 2 13:30:01 c210 systemd: Starting Session 441 of user root. Jun 2 13:40:01 c210 systemd: Started Session 442 of user root. Jun 2 13:40:01 c210 systemd: Starting Session 442 of user root. Jun 2 13:50:01 c210 systemd: Started Session 443 of user root. Jun 2 13:50:01 c210 systemd: Starting Session 443 of user root. Jun 2 14:00:01 c210 systemd: Started Session 444 of user root. Jun 2 14:00:01 c210 systemd: Starting Session 444 of user root. Jun 2 14:01:01 c210 systemd: Started Session 445 of user root. Jun 2 14:01:01 c210 systemd: Starting Session 445 of user root. Jun 2 14:10:01 c210 systemd: Started Session 446 of user root. Jun 2 14:10:01 c210 systemd: Starting Session 446 of user root. Jun 2 14:20:01 c210 systemd: Started Session 447 of user root. Jun 2 14:20:01 c210 systemd: Starting Session 447 of user root. Jun 2 14:30:01 c210 systemd: Started Session 448 of user root. Jun 2 14:30:01 c210 systemd: Starting Session 448 of user root. Jun 2 14:40:01 c210 systemd: Started Session 449 of user root. Jun 2 14:40:01 c210 systemd: Starting Session 449 of user root. Jun 2 14:50:01 c210 systemd: Started Session 450 of user root. Jun 2 14:50:01 c210 systemd: Starting Session 450 of user root. Jun 2 15:00:01 c210 systemd: Started Session 451 of user root. Jun 2 15:00:01 c210 systemd: Starting Session 451 of user root. Jun 2 15:01:01 c210 systemd: Started Session 452 of user root. Jun 2 15:01:01 c210 systemd: Starting Session 452 of user root. Jun 2 15:10:01 c210 systemd: Started Session 453 of user root. Jun 2 15:10:01 c210 systemd: Starting Session 453 of user root. Jun 2 15:20:01 c210 systemd: Started Session 454 of user root. Jun 2 15:20:01 c210 systemd: Starting Session 454 of user root. Jun 2 15:30:01 c210 systemd: Started Session 455 of user root. Jun 2 15:30:01 c210 systemd: Starting Session 455 of user root. Jun 2 15:40:01 c210 systemd: Started Session 456 of user root. Jun 2 15:40:01 c210 systemd: Starting Session 456 of user root. Jun 2 15:50:01 c210 systemd: Started Session 457 of user root. Jun 2 15:50:01 c210 systemd: Starting Session 457 of user root. Jun 2 16:00:01 c210 systemd: Started Session 458 of user root. Jun 2 16:00:01 c210 systemd: Starting Session 458 of user root. Jun 2 16:01:01 c210 systemd: Started Session 459 of user root. Jun 2 16:01:01 c210 systemd: Starting Session 459 of user root. Jun 2 16:10:01 c210 systemd: Started Session 460 of user root. Jun 2 16:10:01 c210 systemd: Starting Session 460 of user root. Jun 2 16:20:01 c210 systemd: Started Session 461 of user root. Jun 2 16:20:01 c210 systemd: Starting Session 461 of user root. Jun 2 16:30:01 c210 systemd: Started Session 462 of user root. Jun 2 16:30:01 c210 systemd: Starting Session 462 of user root. Jun 2 16:40:01 c210 systemd: Started Session 463 of user root. Jun 2 16:40:01 c210 systemd: Starting Session 463 of user root. Jun 2 16:43:35 c210 chronyd[960]: Selected source 45.43.30.59 Jun 2 16:50:01 c210 systemd: Started Session 464 of user root. Jun 2 16:50:01 c210 systemd: Starting Session 464 of user root. Jun 2 17:00:01 c210 systemd: Started Session 465 of user root. Jun 2 17:00:01 c210 systemd: Starting Session 465 of user root. Jun 2 17:01:01 c210 systemd: Started Session 466 of user root. Jun 2 17:01:01 c210 systemd: Starting Session 466 of user root. Jun 2 17:10:01 c210 systemd: Started Session 467 of user root. Jun 2 17:10:01 c210 systemd: Starting Session 467 of user root. Jun 2 17:20:01 c210 systemd: Started Session 468 of user root. Jun 2 17:20:01 c210 systemd: Starting Session 468 of user root. Jun 2 17:30:01 c210 systemd: Started Session 469 of user root. Jun 2 17:30:01 c210 systemd: Starting Session 469 of user root. Jun 2 17:40:01 c210 systemd: Started Session 470 of user root. Jun 2 17:40:01 c210 systemd: Starting Session 470 of user root. Jun 2 17:50:01 c210 systemd: Started Session 471 of user root. Jun 2 17:50:01 c210 systemd: Starting Session 471 of user root. Jun 2 17:52:02 c210 kernel: Lustre: 25173:0:(client.c:2141:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1559519513/real 1559519515] req@ffffa038b0a41680 x1635157931013696/t0(0) o103->es90-OST0000-osc-ffffa04c7febc800@172.16.254.190@o2ib:17/18 lens 3584/224 e 0 to 1 dl 1559519520 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Jun 2 17:52:02 c210 kernel: Lustre: es90-OST0000-osc-ffffa04c7febc800: Connection to es90-OST0000 (at 172.16.254.190@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jun 2 17:52:02 c210 kernel: Lustre: Skipped 7 previous similar messages Jun 2 17:52:05 c210 kernel: Lustre: 25177:0:(client.c:2141:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1559519516/real 1559519518] req@ffffa03fb631d100 x1635157931013984/t0(0) o400->MGC172.16.254.190@o2ib@172.16.254.190@o2ib:26/25 lens 224/224 e 0 to 1 dl 1559519523 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 2 17:52:05 c210 kernel: Lustre: 25177:0:(client.c:2141:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jun 2 17:52:05 c210 kernel: LustreError: 166-1: MGC172.16.254.190@o2ib: Connection to MGS (at 172.16.254.190@o2ib) was lost; in progress operations using this service will fail Jun 2 17:52:05 c210 kernel: LNetError: 25160:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds Jun 2 17:52:05 c210 kernel: LNetError: 25160:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 172.16.254.160@o2ib (7): c: 0, oc: 0, rc: 8 Jun 2 17:52:10 c210 kernel: LNetError: 25161:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) Jun 2 17:52:10 c210 kernel: LNetError: 25161:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Jun 2 17:52:10 c210 kernel: Lustre: 25180:0:(client.c:2141:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1559519519/real 0] req@ffffa03d07c14c80 x1635157931014560/t0(0) o103->es90-OST0000-osc-ffffa04c7febc800@172.16.254.190@o2ib:17/18 lens 3584/224 e 0 to 1 dl 1559519529 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 2 17:52:10 c210 kernel: Lustre: 25180:0:(client.c:2141:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Jun 2 17:52:13 c210 kernel: LNetError: 25161:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) Jun 2 17:52:13 c210 kernel: Lustre: es90-MDT0006-mdc-ffffa04c7febc800: Connection to es90-MDT0006 (at 172.16.254.162@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jun 2 17:52:13 c210 kernel: Lustre: Skipped 12 previous similar messages Jun 2 17:52:16 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 172.16.254.160@o2ib: 13 seconds Jun 2 17:52:16 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 215 previous similar messages Jun 2 17:52:20 c210 kernel: Lustre: 25179:0:(client.c:2141:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1559519525/real 0] req@ffffa040d9af2d00 x1635157931015344/t0(0) o400->es90-MDT0000-mdc-ffffa04c7febc800@172.16.254.190@o2ib:12/10 lens 224/224 e 0 to 1 dl 1559519532 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 2 17:52:20 c210 kernel: Lustre: 25179:0:(client.c:2141:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jun 2 17:52:32 c210 kernel: LustreError: 11290:0:(service.c:2216:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-172.16.254.162@o2ib: deadline 6:1s ago#012 req@ffffa03968bd1200 x1635232869013552/t0(0) o104->LOV_OSC_UUID@172.16.254.162@o2ib:611/0 lens 296/0 e 0 to 0 dl 1559519551 ref 2 fl Interpret:/0/ffffffff rc 0/-1 Jun 2 17:52:32 c210 kernel: Lustre: 11290:0:(service.c:2254:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:1s); client may timeout. req@ffffa03968bd1200 x1635232869013552/t0(0) o104->LOV_OSC_UUID@172.16.254.162@o2ib:611/0 lens 296/0 e 0 to 0 dl 1559519551 ref 2 fl Interpret:/0/ffffffff rc 0/-1 Jun 2 17:52:35 c210 kernel: LustreError: 11292:0:(service.c:2216:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-172.16.254.160@o2ib: deadline 6:2s ago#012 req@ffffa0386d67e300 x1635233215164384/t0(0) o104->LOV_OSC_UUID@172.16.254.160@o2ib:611/0 lens 296/0 e 0 to 0 dl 1559519551 ref 2 fl Interpret:/0/ffffffff rc 0/-1 Jun 2 17:52:35 c210 kernel: LustreError: 11292:0:(service.c:2216:ptlrpc_server_handle_request()) Skipped 501 previous similar messages Jun 2 17:52:35 c210 kernel: Lustre: 11292:0:(service.c:2254:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:2s); client may timeout. req@ffffa0386d67e300 x1635233215164384/t0(0) o104->LOV_OSC_UUID@172.16.254.160@o2ib:611/0 lens 296/0 e 0 to 0 dl 1559519551 ref 2 fl Interpret:/0/ffffffff rc 0/-1 Jun 2 17:52:35 c210 kernel: Lustre: 11292:0:(service.c:2254:ptlrpc_server_handle_request()) Skipped 501 previous similar messages Jun 2 17:52:35 c210 kernel: Lustre: 11333:0:(service.c:2254:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:3s); client may timeout. req@ffffa0378cba6780 x1635232734835040/t0(0) o104->LOV_OSC_UUID@172.16.254.190@o2ib:611/0 lens 296/192 e 0 to 0 dl 1559519551 ref 2 fl Complete:/0/0 rc 0/0 Jun 2 17:52:35 c210 kernel: Lustre: 11333:0:(service.c:2254:ptlrpc_server_handle_request()) Skipped 689 previous similar messages Jun 2 17:52:35 c210 kernel: LustreError: 11333:0:(service.c:2216:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-172.16.254.192@o2ib: deadline 6:3s ago#012 req@ffffa0334935e780 x1635232922335856/t0(0) o104->LOV_OSC_UUID@172.16.254.192@o2ib:611/0 lens 296/0 e 0 to 0 dl 1559519551 ref 2 fl Interpret:/0/ffffffff rc 0/-1 Jun 2 17:52:35 c210 kernel: LustreError: 11333:0:(service.c:2216:ptlrpc_server_handle_request()) Skipped 689 previous similar messages Jun 2 17:52:35 c210 kernel: Lustre: 11284:0:(service.c:1366:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-4s), not sending early reply. Consider increasing at_early_margin (5)? req@ffffa03792b63600 x1635232921622608/t0(0) o104->LOV_OSC_UUID@172.16.254.192@o2ib:611/0 lens 296/0 e 0 to 0 dl 1559519551 ref 2 fl New:/2/ffffffff rc 0/-1 Jun 2 17:52:36 c210 kernel: LustreError: 11290:0:(service.c:2216:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-172.16.254.191@o2ib: deadline 6:5s ago#012 req@ffffa035793f9200 x1635232884665680/t0(0) o104->LOV_OSC_UUID@172.16.254.191@o2ib:611/0 lens 296/0 e 0 to 0 dl 1559519551 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jun 2 17:52:36 c210 kernel: LustreError: 11290:0:(service.c:2216:ptlrpc_server_handle_request()) Skipped 3317 previous similar messages Jun 2 17:52:36 c210 kernel: Lustre: 11290:0:(service.c:2254:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:5s); client may timeout. req@ffffa035793f9200 x1635232884665680/t0(0) o104->LOV_OSC_UUID@172.16.254.191@o2ib:611/0 lens 296/0 e 0 to 0 dl 1559519551 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Jun 2 17:52:36 c210 kernel: Lustre: 11290:0:(service.c:2254:ptlrpc_server_handle_request()) Skipped 3318 previous similar messages Jun 2 17:53:20 c210 kernel: Lustre: MGC172.16.254.190@o2ib: Connection restored to 172.16.254.190@o2ib (at 172.16.254.190@o2ib) Jun 2 17:53:20 c210 kernel: Lustre: Skipped 7 previous similar messages Jun 2 17:53:42 c210 kernel: Lustre: 6327:0:(client.c:2141:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1559519611/real 1559519612] req@ffffa04b5872da00 x1635157931055392/t0(0) o101->MGC172.16.254.190@o2ib@172.16.254.190@o2ib:26/25 lens 328/344 e 0 to 1 dl 1559519621 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 2 17:53:42 c210 kernel: Lustre: 6327:0:(client.c:2141:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jun 2 17:53:42 c210 kernel: LustreError: 166-1: MGC172.16.254.190@o2ib: Connection to MGS (at 172.16.254.190@o2ib) was lost; in progress operations using this service will fail Jun 2 17:53:45 c210 kernel: LNetError: 22059:0:(o2iblnd_cb.c:3307:kiblnd_cm_callback()) 172.16.254.161@o2ib DISCONNECTED Jun 2 17:53:47 c210 kernel: Lustre: es90-MDT0007-mdc-ffffa04c7febc800: Connection restored to 172.16.254.163@o2ib (at 172.16.254.163@o2ib) Jun 2 17:53:47 c210 kernel: LustreError: 167-0: es90-OST000e-osc-ffffa04c7febc800: This client was evicted by es90-OST000e; in progress operations using this service will fail. Jun 2 17:53:47 c210 kernel: LustreError: Skipped 7 previous similar messages Jun 2 17:54:06 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 172.16.254.161@o2ib: 21 seconds Jun 2 17:54:06 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 78 previous similar messages Jun 2 17:54:18 c210 kernel: LustreError: 167-0: es90-OST000f-osc-ffffa04c7febc800: This client was evicted by es90-OST000f; in progress operations using this service will fail. Jun 2 17:54:24 c210 kernel: LNetError: 21:0:(o2iblnd_cb.c:3307:kiblnd_cm_callback()) 172.16.254.191@o2ib DISCONNECTED Jun 2 17:54:44 c210 kernel: LNetError: 25163:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) Jun 2 17:54:44 c210 kernel: LNetError: 25163:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 7 previous similar messages Jun 2 17:54:51 c210 kernel: Lustre: 25180:0:(client.c:2141:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1559519675/real 0] req@ffffa0306132de80 x1635157931064384/t0(0) o400->es90-MDT0007-mdc-ffffa04c7febc800@172.16.254.163@o2ib:12/10 lens 224/224 e 0 to 1 dl 1559519686 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 2 17:54:51 c210 kernel: Lustre: es90-MDT0007-mdc-ffffa04c7febc800: Connection to es90-MDT0007 (at 172.16.254.163@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jun 2 17:54:51 c210 kernel: Lustre: Skipped 10 previous similar messages Jun 2 17:55:16 c210 kernel: NMI watchdog: BUG: soft lockup - CPU#9 stuck for 21s! [ll_imp_inval:4395] Jun 2 17:55:16 c210 kernel: Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper iTCO_wdt cryptd joydev iTCO_vendor_support sg ipmi_si shpchp ipmi_devintf wmi mei_me mei ioatdma ipmi_msghandler i2c_i801 pcspkr lpc_ich binfmt_misc knem(OE) ip_tables ext4 mbcache jbd2 mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit Jun 2 17:55:16 c210 kernel: drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ixgbe drm libahci libata mlx5_core(OE) crct10dif_pclmul crct10dif_common crc32c_intel megaraid_sas mlxfw(OE) devlink mdio i2c_core mlx_compat(OE) ptp pps_core dca dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Jun 2 17:55:16 c210 kernel: CPU: 9 PID: 4395 Comm: ll_imp_inval Kdump: loaded Tainted: G OEL ------------ 3.10.0-862.el7.x86_64 #1 Jun 2 17:55:16 c210 kernel: Hardware name: Supermicro SYS-1027R-WC1RT/X9DRW-CF/CTF, BIOS 3.0b 04/09/2014 Jun 2 17:55:16 c210 kernel: task: ffffa03cd6b68fd0 ti: ffffa04551238000 task.ti: ffffa04551238000 Jun 2 17:55:16 c210 kernel: RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 Jun 2 17:55:16 c210 kernel: RSP: 0018:ffffa0455123ba90 EFLAGS: 00000246 Jun 2 17:55:16 c210 kernel: RAX: 0000000000000000 RBX: ffffa04c27136640 RCX: 0000000000490000 Jun 2 17:55:16 c210 kernel: RDX: ffffa03c3fcd9700 RSI: 0000000000190001 RDI: ffffa045b18e4948 Jun 2 17:55:16 c210 kernel: RBP: ffffa0455123ba90 R08: ffffa04c7fc59700 R09: 0000000000000000 Jun 2 17:55:16 c210 kernel: R10: 0000000000000000 R11: fffff0189d75c0c0 R12: ffffa0455123ba58 Jun 2 17:55:16 c210 kernel: R13: ffffa02e77788000 R14: ffffa03791272ea8 R15: ffffa02e777880e8 Jun 2 17:55:16 c210 kernel: FS: 0000000000000000(0000) GS:ffffa04c7fc40000(0000) knlGS:0000000000000000 Jun 2 17:55:16 c210 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 2 17:55:16 c210 kernel: CR2: 00007f1002906000 CR3: 000000084a20e000 CR4: 00000000001607e0 Jun 2 17:55:16 c210 kernel: Call Trace: Jun 2 17:55:16 c210 kernel: [] queued_spin_lock_slowpath+0xb/0xf Jun 2 17:55:16 c210 kernel: [] _raw_spin_lock+0x20/0x30 Jun 2 17:55:16 c210 kernel: [] cl_object_attr_lock+0x1a/0x20 [obdclass] Jun 2 17:55:16 c210 kernel: [] osc_ldlm_blocking_ast+0x2f6/0x3a0 [osc] Jun 2 17:55:16 c210 kernel: [] ldlm_cancel_callback+0x8a/0x330 [ptlrpc] Jun 2 17:55:16 c210 kernel: [] ? class_handle_unhash+0x32/0x50 [obdclass] Jun 2 17:55:16 c210 kernel: [] ? __slab_free+0x81/0x2f0 Jun 2 17:55:16 c210 kernel: [] ldlm_cli_cancel_local+0xa0/0x3f0 [ptlrpc] Jun 2 17:55:16 c210 kernel: [] ldlm_cli_cancel+0x381/0x650 [ptlrpc] Jun 2 17:55:16 c210 kernel: [] cleanup_resource+0x1f9/0x330 [ptlrpc] Jun 2 17:55:16 c210 kernel: [] ldlm_resource_clean+0x35/0x50 [ptlrpc] Jun 2 17:55:16 c210 kernel: [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] Jun 2 17:55:16 c210 kernel: [] ? cleanup_resource+0x330/0x330 [ptlrpc] Jun 2 17:55:16 c210 kernel: [] ? cleanup_resource+0x330/0x330 [ptlrpc] Jun 2 17:55:16 c210 kernel: [] cfs_hash_for_each_nolock+0x75/0x1c0 [libcfs] Jun 2 17:55:16 c210 kernel: [] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] Jun 2 17:55:16 c210 kernel: [] osc_import_event+0x459/0xfc0 [osc] Jun 2 17:55:16 c210 kernel: [] ? ptlrpc_import_recovery_state_machine+0xc20/0xc20 [ptlrpc] Jun 2 17:55:16 c210 kernel: [] ptlrpc_invalidate_import+0x2a4/0xae0 [ptlrpc] Jun 2 17:55:16 c210 kernel: [] ? ptlrpc_import_recovery_state_machine+0xc20/0xc20 [ptlrpc] Jun 2 17:55:16 c210 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Jun 2 17:55:16 c210 kernel: [] ? ptlrpc_import_recovery_state_machine+0xc20/0xc20 [ptlrpc] Jun 2 17:55:16 c210 kernel: [] ptlrpc_invalidate_import_thread+0x48/0x2b0 [ptlrpc] Jun 2 17:55:16 c210 kernel: [] kthread+0xd1/0xe0 Jun 2 17:55:16 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 17:55:16 c210 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Jun 2 17:55:16 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 17:55:16 c210 kernel: Code: 0d 48 98 83 e2 30 48 81 c2 00 97 01 00 48 03 14 c5 a0 53 73 87 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 Jun 2 17:55:16 c210 kernel: LNetError: 25160:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: active_txs, 11 seconds Jun 2 17:55:16 c210 kernel: LNetError: 25160:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Skipped 7 previous similar messages Jun 2 17:55:16 c210 kernel: LNetError: 25160:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 172.16.254.163@o2ib (16): c: 4, oc: 3, rc: 8 Jun 2 17:55:16 c210 kernel: LNetError: 25160:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Skipped 7 previous similar messages Jun 2 17:55:25 c210 kernel: LustreError: 6327:0:(mgc_request.c:599:do_requeue()) failed processing log: -5 Jun 2 17:55:30 c210 kernel: NMI watchdog: BUG: soft lockup - CPU#9 stuck for 24s! [ldlm_bl_04:9838] Jun 2 17:55:30 c210 kernel: Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper iTCO_wdt cryptd joydev iTCO_vendor_support sg ipmi_si shpchp ipmi_devintf wmi mei_me mei ioatdma ipmi_msghandler i2c_i801 pcspkr lpc_ich binfmt_misc knem(OE) ip_tables ext4 mbcache jbd2 mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit Jun 2 17:55:30 c210 kernel: drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ixgbe drm libahci libata mlx5_core(OE) crct10dif_pclmul crct10dif_common crc32c_intel megaraid_sas mlxfw(OE) devlink mdio i2c_core mlx_compat(OE) ptp pps_core dca dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Jun 2 17:55:30 c210 kernel: CPU: 9 PID: 9838 Comm: ldlm_bl_04 Kdump: loaded Tainted: G OEL ------------ 3.10.0-862.el7.x86_64 #1 Jun 2 17:55:30 c210 kernel: Hardware name: Supermicro SYS-1027R-WC1RT/X9DRW-CF/CTF, BIOS 3.0b 04/09/2014 Jun 2 17:55:30 c210 kernel: task: ffffa03c3f2d8fd0 ti: ffffa0305561c000 task.ti: ffffa0305561c000 Jun 2 17:55:30 c210 kernel: RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 Jun 2 17:55:30 c210 kernel: RSP: 0018:ffffa0305561fb48 EFLAGS: 00000246 Jun 2 17:55:30 c210 kernel: RAX: 0000000000000000 RBX: ffffffffc0f84b7f RCX: 0000000000490000 Jun 2 17:55:30 c210 kernel: RDX: ffffa04c7fcd9700 RSI: 0000000000590001 RDI: ffffa045b18e4948 Jun 2 17:55:30 c210 kernel: RBP: ffffa0305561fb48 R08: ffffa04c7fc59700 R09: 0000000000000000 Jun 2 17:55:30 c210 kernel: R10: 0000000000000000 R11: fffff018d0d22d00 R12: ffffa03dfa028b28 Jun 2 17:55:30 c210 kernel: R13: ffffa0468e353ab8 R14: ffffa03dfa028c10 R15: 000000000008a339 Jun 2 17:55:30 c210 kernel: FS: 0000000000000000(0000) GS:ffffa04c7fc40000(0000) knlGS:0000000000000000 Jun 2 17:55:30 c210 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 2 17:55:30 c210 kernel: CR2: 00007fd70bd2e000 CR3: 000000203b354000 CR4: 00000000001607e0 Jun 2 17:55:30 c210 kernel: Call Trace: Jun 2 17:55:30 c210 kernel: [] queued_spin_lock_slowpath+0xb/0xf Jun 2 17:55:30 c210 kernel: [] _raw_spin_lock+0x20/0x30 Jun 2 17:55:30 c210 kernel: [] cl_object_attr_lock+0x1a/0x20 [obdclass] Jun 2 17:55:51 c210 kernel: [] osc_ldlm_blocking_ast+0x2f6/0x3a0 [osc] Jun 2 17:55:51 c210 kernel: [] ldlm_cancel_callback+0x8a/0x330 [ptlrpc] Jun 2 17:55:51 c210 kernel: [] ? class_handle_unhash+0x32/0x50 [obdclass] Jun 2 17:55:51 c210 kernel: [] ldlm_cli_cancel_local+0xa0/0x3f0 [ptlrpc] Jun 2 17:55:51 c210 kernel: [] ldlm_cli_cancel_list_local+0xea/0x280 [ptlrpc] Jun 2 17:55:51 c210 kernel: [] ldlm_cancel_lru_local+0x2b/0x30 [ptlrpc] Jun 2 17:55:51 c210 kernel: [] ldlm_cli_cancel+0x216/0x650 [ptlrpc] Jun 2 17:55:51 c210 kernel: [] osc_ldlm_blocking_ast+0x17a/0x3a0 [osc] Jun 2 17:55:51 c210 kernel: [] ldlm_handle_bl_callback+0xed/0x4e0 [ptlrpc] Jun 2 17:55:51 c210 kernel: [] ldlm_bl_thread_main+0x531/0x700 [ptlrpc] Jun 2 17:55:51 c210 kernel: [] ? wake_up_state+0x20/0x20 Jun 2 17:55:51 c210 kernel: [] ? ldlm_handle_bl_callback+0x4e0/0x4e0 [ptlrpc] Jun 2 17:55:51 c210 kernel: [] kthread+0xd1/0xe0 Jun 2 17:55:51 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 17:55:51 c210 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Jun 2 17:55:51 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 17:55:51 c210 kernel: Code: 0d 48 98 83 e2 30 48 81 c2 00 97 01 00 48 03 14 c5 a0 53 73 87 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 Jun 2 17:57:09 c210 kernel: LustreError: 6327:0:(mgc_request.c:599:do_requeue()) failed processing log: -5 Jun 2 17:57:09 c210 kernel: LNet: 25160:0:(o2iblnd_cb.c:1510:kiblnd_reconnect_peer()) Abort reconnection of 172.16.254.190@o2ib: connected Jun 2 17:57:28 c210 kernel: Lustre: Evicted from MGS (at 172.16.254.190@o2ib) after server handle changed from 0x57b6ec32f8296e7 to 0x57b6ec361b9d881 Jun 2 17:57:32 c210 kernel: Lustre: MGC172.16.254.190@o2ib: Connection restored to 172.16.254.190@o2ib (at 172.16.254.190@o2ib) Jun 2 17:58:16 c210 kernel: NMI watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [ll_imp_inval:4395] Jun 2 17:58:16 c210 kernel: Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper iTCO_wdt cryptd joydev iTCO_vendor_support sg ipmi_si shpchp ipmi_devintf wmi mei_me mei ioatdma ipmi_msghandler i2c_i801 pcspkr lpc_ich binfmt_misc knem(OE) ip_tables ext4 mbcache jbd2 mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit Jun 2 17:58:16 c210 kernel: drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ixgbe drm libahci libata mlx5_core(OE) crct10dif_pclmul crct10dif_common crc32c_intel megaraid_sas mlxfw(OE) devlink mdio i2c_core mlx_compat(OE) ptp pps_core dca dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Jun 2 17:58:16 c210 kernel: CPU: 2 PID: 4395 Comm: ll_imp_inval Kdump: loaded Tainted: G OEL ------------ 3.10.0-862.el7.x86_64 #1 Jun 2 17:58:16 c210 kernel: Hardware name: Supermicro SYS-1027R-WC1RT/X9DRW-CF/CTF, BIOS 3.0b 04/09/2014 Jun 2 17:58:16 c210 kernel: task: ffffa03cd6b68fd0 ti: ffffa04551238000 task.ti: ffffa04551238000 Jun 2 17:58:16 c210 kernel: RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 Jun 2 17:58:16 c210 kernel: RSP: 0018:ffffa0455123ba90 EFLAGS: 00000246 Jun 2 17:58:16 c210 kernel: RAX: 0000000000000000 RBX: ffffa04c27137a40 RCX: 0000000000110000 Jun 2 17:58:16 c210 kernel: RDX: ffffa04c7fd19700 RSI: 0000000000610001 RDI: ffffa045b18e4948 Jun 2 17:58:16 c210 kernel: RBP: ffffa0455123ba90 R08: ffffa03c3fc99700 R09: 0000000000000000 Jun 2 17:58:16 c210 kernel: R10: 0000000000000000 R11: fffff0188dfaac00 R12: ffffa0455123ba58 Jun 2 17:58:16 c210 kernel: R13: ffffa040a5c9b7c8 R14: ffffa03f8d5a13f8 R15: ffffa040a5c9b8b0 Jun 2 17:58:16 c210 kernel: FS: 0000000000000000(0000) GS:ffffa03c3fc80000(0000) knlGS:0000000000000000 Jun 2 17:58:16 c210 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 2 17:58:16 c210 kernel: CR2: 00007f1002906000 CR3: 000000203b354000 CR4: 00000000001607e0 Jun 2 17:58:16 c210 kernel: Call Trace: Jun 2 17:58:16 c210 kernel: [] queued_spin_lock_slowpath+0xb/0xf Jun 2 17:58:16 c210 kernel: [] _raw_spin_lock+0x20/0x30 Jun 2 17:58:16 c210 kernel: [] cl_object_attr_lock+0x1a/0x20 [obdclass] Jun 2 17:58:16 c210 kernel: [] osc_ldlm_blocking_ast+0x2f6/0x3a0 [osc] Jun 2 17:58:16 c210 kernel: [] ldlm_cancel_callback+0x8a/0x330 [ptlrpc] Jun 2 17:58:16 c210 kernel: [] ? class_handle_unhash+0x32/0x50 [obdclass] Jun 2 17:58:16 c210 kernel: [] ? __slab_free+0x81/0x2f0 Jun 2 17:58:16 c210 kernel: [] ldlm_cli_cancel_local+0xa0/0x3f0 [ptlrpc] Jun 2 17:58:16 c210 kernel: [] ldlm_cli_cancel+0x381/0x650 [ptlrpc] Jun 2 17:58:16 c210 kernel: [] cleanup_resource+0x1f9/0x330 [ptlrpc] Jun 2 17:58:16 c210 kernel: [] ldlm_resource_clean+0x35/0x50 [ptlrpc] Jun 2 17:58:16 c210 kernel: [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] Jun 2 17:58:16 c210 kernel: [] ? cleanup_resource+0x330/0x330 [ptlrpc] Jun 2 17:58:16 c210 kernel: [] ? cleanup_resource+0x330/0x330 [ptlrpc] Jun 2 17:58:16 c210 kernel: [] cfs_hash_for_each_nolock+0x75/0x1c0 [libcfs] Jun 2 17:58:16 c210 kernel: [] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] Jun 2 17:58:16 c210 kernel: [] osc_import_event+0x459/0xfc0 [osc] Jun 2 17:58:16 c210 kernel: [] ? ptlrpc_import_recovery_state_machine+0xc20/0xc20 [ptlrpc] Jun 2 17:58:16 c210 kernel: [] ptlrpc_invalidate_import+0x2a4/0xae0 [ptlrpc] Jun 2 17:58:16 c210 kernel: [] ? ptlrpc_import_recovery_state_machine+0xc20/0xc20 [ptlrpc] Jun 2 17:58:16 c210 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Jun 2 17:58:16 c210 kernel: [] ? ptlrpc_import_recovery_state_machine+0xc20/0xc20 [ptlrpc] Jun 2 17:58:16 c210 kernel: [] ptlrpc_invalidate_import_thread+0x48/0x2b0 [ptlrpc] Jun 2 17:58:16 c210 kernel: [] kthread+0xd1/0xe0 Jun 2 17:58:16 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 17:58:16 c210 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Jun 2 17:58:16 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 17:58:16 c210 kernel: Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 00 97 01 00 48 03 14 c5 a0 53 73 87 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b Jun 2 17:58:39 c210 kernel: LustreError: 4394:0:(ldlm_resource.c:1146:ldlm_resource_complain()) es90-OST000e-osc-ffffa04c7febc800: namespace resource [0x9c620:0x0:0x0].0x0 (ffffa0446b8632c0) refcount nonzero (1062) after lock cleanup; forcing cleanup. Jun 2 17:58:39 c210 kernel: LustreError: 4394:0:(ldlm_resource.c:1146:ldlm_resource_complain()) Skipped 55 previous similar messages Jun 2 17:58:39 c210 kernel: Lustre: es90-OST000e-osc-ffffa04c7febc800: Connection restored to 172.16.254.163@o2ib (at 172.16.254.163@o2ib) Jun 2 17:58:46 c210 kernel: Lustre: es90-OST000e-osc-ffffa04c7febc800: Connection to es90-OST000e (at 172.16.254.163@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jun 2 17:58:46 c210 kernel: LustreError: 167-0: es90-OST000e-osc-ffffa04c7febc800: This client was evicted by es90-OST000e; in progress operations using this service will fail. Jun 2 17:58:46 c210 kernel: LustreError: 4457:0:(ldlm_resource.c:1146:ldlm_resource_complain()) es90-OST000e-osc-ffffa04c7febc800: namespace resource [0x9c620:0x0:0x0].0x0 (ffffa0446b8632c0) refcount nonzero (1062) after lock cleanup; forcing cleanup. Jun 2 17:58:46 c210 kernel: LustreError: 4457:0:(ldlm_resource.c:1146:ldlm_resource_complain()) Skipped 15 previous similar messages Jun 2 18:00:15 c210 kernel: NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ll_imp_inval:4395] Jun 2 18:00:15 c210 kernel: Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper iTCO_wdt cryptd joydev iTCO_vendor_support sg ipmi_si shpchp ipmi_devintf wmi mei_me mei ioatdma ipmi_msghandler i2c_i801 pcspkr lpc_ich binfmt_misc knem(OE) ip_tables ext4 mbcache jbd2 mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit Jun 2 18:00:15 c210 kernel: drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ixgbe drm libahci libata mlx5_core(OE) crct10dif_pclmul crct10dif_common crc32c_intel megaraid_sas mlxfw(OE) devlink mdio i2c_core mlx_compat(OE) ptp pps_core dca dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Jun 2 18:00:15 c210 kernel: CPU: 0 PID: 4395 Comm: ll_imp_inval Kdump: loaded Tainted: G OEL ------------ 3.10.0-862.el7.x86_64 #1 Jun 2 18:00:15 c210 kernel: Hardware name: Supermicro SYS-1027R-WC1RT/X9DRW-CF/CTF, BIOS 3.0b 04/09/2014 Jun 2 18:00:15 c210 kernel: task: ffffa03cd6b68fd0 ti: ffffa04551238000 task.ti: ffffa04551238000 Jun 2 18:00:15 c210 kernel: RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 Jun 2 18:00:15 c210 kernel: RSP: 0018:ffffa0455123ba90 EFLAGS: 00000246 Jun 2 18:00:15 c210 kernel: RAX: 0000000000000000 RBX: ffffa0457462c3c0 RCX: 0000000000010000 Jun 2 18:00:15 c210 kernel: RDX: ffffa04c7fc99700 RSI: 0000000000510001 RDI: ffffa045b18e4948 Jun 2 18:00:15 c210 kernel: RBP: ffffa0455123ba90 R08: ffffa03c3fc19700 R09: 0000000000000000 Jun 2 18:00:15 c210 kernel: R10: 0000000000000000 R11: fffff018fbf428c0 R12: ffffa0455123ba58 Jun 2 18:00:15 c210 kernel: R13: ffffa040a5c9b7c8 R14: ffffa03f8d5a13f8 R15: ffffa040a5c9b8b0 Jun 2 18:00:15 c210 kernel: FS: 0000000000000000(0000) GS:ffffa03c3fc00000(0000) knlGS:0000000000000000 Jun 2 18:00:15 c210 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 2 18:00:15 c210 kernel: CR2: 00007fba59f2f000 CR3: 000000203be54000 CR4: 00000000001607f0 Jun 2 18:00:15 c210 kernel: Call Trace: Jun 2 18:00:15 c210 kernel: [] queued_spin_lock_slowpath+0xb/0xf Jun 2 18:00:15 c210 kernel: [] _raw_spin_lock+0x20/0x30 Jun 2 18:00:15 c210 kernel: [] cl_object_attr_lock+0x1a/0x20 [obdclass] Jun 2 18:00:15 c210 kernel: [] osc_ldlm_blocking_ast+0x2f6/0x3a0 [osc] Jun 2 18:00:15 c210 kernel: [] ldlm_cancel_callback+0x8a/0x330 [ptlrpc] Jun 2 18:00:15 c210 kernel: [] ? class_handle_unhash+0x32/0x50 [obdclass] Jun 2 18:00:15 c210 kernel: [] ? __slab_free+0x81/0x2f0 Jun 2 18:00:15 c210 kernel: [] ldlm_cli_cancel_local+0xa0/0x3f0 [ptlrpc] Jun 2 18:00:15 c210 kernel: [] ldlm_cli_cancel+0x381/0x650 [ptlrpc] Jun 2 18:00:15 c210 kernel: [] cleanup_resource+0x1f9/0x330 [ptlrpc] Jun 2 18:00:15 c210 kernel: [] ldlm_resource_clean+0x35/0x50 [ptlrpc] Jun 2 18:00:15 c210 kernel: [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] Jun 2 18:00:15 c210 kernel: [] ? cleanup_resource+0x330/0x330 [ptlrpc] Jun 2 18:00:15 c210 kernel: [] ? cleanup_resource+0x330/0x330 [ptlrpc] Jun 2 18:00:15 c210 kernel: [] cfs_hash_for_each_nolock+0x75/0x1c0 [libcfs] Jun 2 18:00:15 c210 kernel: [] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] Jun 2 18:00:15 c210 kernel: [] osc_import_event+0x459/0xfc0 [osc] Jun 2 18:00:15 c210 kernel: [] ? ptlrpc_import_recovery_state_machine+0xc20/0xc20 [ptlrpc] Jun 2 18:00:15 c210 kernel: [] ptlrpc_invalidate_import+0x2a4/0xae0 [ptlrpc] Jun 2 18:00:15 c210 kernel: [] ? ptlrpc_import_recovery_state_machine+0xc20/0xc20 [ptlrpc] Jun 2 18:00:15 c210 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Jun 2 18:00:15 c210 kernel: [] ? ptlrpc_import_recovery_state_machine+0xc20/0xc20 [ptlrpc] Jun 2 18:00:15 c210 kernel: [] ptlrpc_invalidate_import_thread+0x48/0x2b0 [ptlrpc] Jun 2 18:00:15 c210 kernel: [] kthread+0xd1/0xe0 Jun 2 18:00:15 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:00:15 c210 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Jun 2 18:00:15 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:00:15 c210 kernel: Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 00 97 01 00 48 03 14 c5 a0 53 73 87 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b Jun 2 18:00:20 c210 systemd: Started Session 472 of user root. Jun 2 18:00:20 c210 systemd: Starting Session 472 of user root. Jun 2 18:00:54 c210 kernel: NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ldlm_bl_43:4442] Jun 2 18:00:54 c210 kernel: Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) sunrpc sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper iTCO_wdt cryptd joydev iTCO_vendor_support sg ipmi_si shpchp ipmi_devintf wmi mei_me mei ioatdma ipmi_msghandler i2c_i801 pcspkr lpc_ich binfmt_misc knem(OE) ip_tables ext4 mbcache jbd2 mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit Jun 2 18:00:54 c210 kernel: drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ixgbe drm libahci libata mlx5_core(OE) crct10dif_pclmul crct10dif_common crc32c_intel megaraid_sas mlxfw(OE) devlink mdio i2c_core mlx_compat(OE) ptp pps_core dca dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Jun 2 18:00:54 c210 kernel: CPU: 0 PID: 4442 Comm: ldlm_bl_43 Kdump: loaded Tainted: G OEL ------------ 3.10.0-862.el7.x86_64 #1 Jun 2 18:00:54 c210 kernel: Hardware name: Supermicro SYS-1027R-WC1RT/X9DRW-CF/CTF, BIOS 3.0b 04/09/2014 Jun 2 18:00:54 c210 kernel: task: ffffa04c3bb3eeb0 ti: ffffa031b58a0000 task.ti: ffffa031b58a0000 Jun 2 18:00:54 c210 kernel: RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 Jun 2 18:00:54 c210 kernel: RSP: 0018:ffffa031b58a3b48 EFLAGS: 00000246 Jun 2 18:00:54 c210 kernel: RAX: 0000000000000000 RBX: ffffffffc0f84b7f RCX: 0000000000010000 Jun 2 18:00:54 c210 kernel: RDX: ffffa04c7fcd9700 RSI: 0000000000590001 RDI: ffffa045b18e4948 Jun 2 18:00:54 c210 kernel: RBP: ffffa031b58a3b48 R08: ffffa03c3fc19700 R09: 0000000000000000 Jun 2 18:00:54 c210 kernel: R10: 0000000000000000 R11: fffff018b1d64480 R12: ffffa040a5c9b7c8 Jun 2 18:00:54 c210 kernel: R13: ffffa03f8d5a13f8 R14: ffffa040a5c9b8b0 R15: 00000000000a5aff Jun 2 18:00:54 c210 kernel: FS: 0000000000000000(0000) GS:ffffa03c3fc00000(0000) knlGS:0000000000000000 Jun 2 18:00:54 c210 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 2 18:00:54 c210 kernel: CR2: 00007fd700006bc9 CR3: 000000203be54000 CR4: 00000000001607f0 Jun 2 18:00:54 c210 kernel: Call Trace: Jun 2 18:00:54 c210 kernel: [] queued_spin_lock_slowpath+0xb/0xf Jun 2 18:00:54 c210 kernel: [] _raw_spin_lock+0x20/0x30 Jun 2 18:00:54 c210 kernel: [] cl_object_attr_lock+0x1a/0x20 [obdclass] Jun 2 18:00:54 c210 kernel: [] osc_ldlm_blocking_ast+0x2f6/0x3a0 [osc] Jun 2 18:00:54 c210 kernel: [] ldlm_cancel_callback+0x8a/0x330 [ptlrpc] Jun 2 18:00:54 c210 kernel: [] ? class_handle_unhash+0x32/0x50 [obdclass] Jun 2 18:00:54 c210 kernel: [] ldlm_cli_cancel_local+0xa0/0x3f0 [ptlrpc] Jun 2 18:00:54 c210 kernel: [] ldlm_cli_cancel_list_local+0xea/0x280 [ptlrpc] Jun 2 18:00:54 c210 kernel: [] ldlm_cancel_lru_local+0x2b/0x30 [ptlrpc] Jun 2 18:00:54 c210 kernel: [] ldlm_cli_cancel+0x216/0x650 [ptlrpc] Jun 2 18:00:54 c210 kernel: [] ? __slab_free+0x81/0x2f0 Jun 2 18:00:54 c210 kernel: [] osc_ldlm_blocking_ast+0x17a/0x3a0 [osc] Jun 2 18:00:54 c210 kernel: [] ldlm_handle_bl_callback+0xed/0x4e0 [ptlrpc] Jun 2 18:00:54 c210 kernel: [] ldlm_bl_thread_main+0x531/0x700 [ptlrpc] Jun 2 18:00:54 c210 kernel: [] ? ldlm_handle_bl_callback+0x4e0/0x4e0 [ptlrpc] Jun 2 18:00:54 c210 kernel: [] kthread+0xd1/0xe0 Jun 2 18:00:54 c210 kernel: [] ? finish_task_switch+0x57/0x170 Jun 2 18:00:54 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:00:54 c210 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Jun 2 18:00:54 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:00:54 c210 kernel: Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 00 97 01 00 48 03 14 c5 a0 53 73 87 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b Jun 2 18:01:46 c210 dbus[854]: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 35123ms) Jun 2 18:01:46 c210 kernel: LustreError: 4395:0:(ldlm_resource.c:1146:ldlm_resource_complain()) es90-OST000f-osc-ffffa04c7febc800: namespace resource [0x9c660:0x0:0x0].0x0 (ffffa04bb3abd380) refcount nonzero (1137) after lock cleanup; forcing cleanup. Jun 2 18:01:46 c210 kernel: LustreError: 4395:0:(ldlm_resource.c:1146:ldlm_resource_complain()) Skipped 15 previous similar messages Jun 2 18:01:46 c210 kernel: Lustre: es90-OST000f-osc-ffffa04c7febc800: Connection restored to 172.16.254.163@o2ib (at 172.16.254.163@o2ib) Jun 2 18:01:46 c210 kernel: Lustre: Skipped 1 previous similar message Jun 2 18:02:01 c210 kernel: Lustre: es90-OST000f-osc-ffffa04c7febc800: Connection to es90-OST000f (at 172.16.254.163@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jun 2 18:02:42 c210 kernel: Lustre: 25180:0:(client.c:2141:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1559520121/real 1559520161] req@ffffa046ea6ec800 x1635157931576272/t0(0) o400->es90-OST000f-osc-ffffa04c7febc800@172.16.254.163@o2ib:28/4 lens 224/224 e 0 to 1 dl 1559520162 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1 Jun 2 18:02:42 c210 kernel: Lustre: 25180:0:(client.c:2141:ptlrpc_expire_one_request()) Skipped 17151 previous similar messages Jun 2 18:03:04 c210 systemd: Started Session 474 of user root. Jun 2 18:03:04 c210 systemd-logind: New session 474 of user root. Jun 2 18:03:04 c210 systemd: Starting Session 474 of user root. Jun 2 18:03:06 c210 systemd-logind: Removed session 474. Jun 2 18:05:09 c210 systemd-logind: New session 475 of user root. Jun 2 18:05:09 c210 systemd: Started Session 475 of user root. Jun 2 18:05:09 c210 systemd: Starting Session 475 of user root. Jun 2 18:05:09 c210 systemd-logind: Removed session 475. Jun 2 18:05:12 c210 systemd-logind: New session 476 of user root. Jun 2 18:05:12 c210 systemd: Started Session 476 of user root. Jun 2 18:05:12 c210 systemd: Starting Session 476 of user root. Jun 2 18:05:12 c210 systemd-logind: Removed session 476. Jun 2 18:06:50 c210 kernel: INFO: task jbd2/dm-0-8:536 blocked for more than 120 seconds. Jun 2 18:06:50 c210 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 2 18:06:50 c210 kernel: jbd2/dm-0-8 D ffffa03c3bc5bf40 0 536 2 0x00000000 Jun 2 18:06:50 c210 kernel: Call Trace: Jun 2 18:06:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:06:50 c210 kernel: [] schedule+0x29/0x70 Jun 2 18:06:50 c210 kernel: [] schedule_timeout+0x239/0x2c0 Jun 2 18:06:50 c210 kernel: [] ? __blk_run_queue+0x39/0x50 Jun 2 18:06:50 c210 kernel: [] ? ktime_get_ts64+0x52/0xf0 Jun 2 18:06:50 c210 kernel: [] ? delayacct_end+0x8f/0xb0 Jun 2 18:06:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:06:50 c210 kernel: [] io_schedule_timeout+0xad/0x130 Jun 2 18:06:50 c210 kernel: [] io_schedule+0x18/0x20 Jun 2 18:06:50 c210 kernel: [] bit_wait_io+0x11/0x50 Jun 2 18:06:50 c210 kernel: [] __wait_on_bit+0x67/0x90 Jun 2 18:06:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:06:50 c210 kernel: [] out_of_line_wait_on_bit+0x81/0xb0 Jun 2 18:06:50 c210 kernel: [] ? wake_bit_function+0x40/0x40 Jun 2 18:06:50 c210 kernel: [] __wait_on_buffer+0x2a/0x30 Jun 2 18:06:50 c210 kernel: [] jbd2_journal_commit_transaction+0x1781/0x19b0 [jbd2] Jun 2 18:06:50 c210 kernel: [] kjournald2+0xc9/0x260 [jbd2] Jun 2 18:06:50 c210 kernel: [] ? wake_up_atomic_t+0x30/0x30 Jun 2 18:06:50 c210 kernel: [] ? commit_timeout+0x10/0x10 [jbd2] Jun 2 18:06:50 c210 kernel: [] kthread+0xd1/0xe0 Jun 2 18:06:50 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:06:50 c210 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Jun 2 18:06:50 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:07:11 c210 systemd-logind: New session 477 of user root. Jun 2 18:07:11 c210 systemd: Started Session 477 of user root. Jun 2 18:07:11 c210 systemd: Starting Session 477 of user root. Jun 2 18:07:11 c210 systemd-logind: Removed session 477. Jun 2 18:07:28 c210 kernel: LustreError: 11-0: es90-OST000e-osc-ffffa04c7febc800: operation ost_set_info to node 172.16.254.163@o2ib failed: rc = -107 Jun 2 18:07:28 c210 kernel: LustreError: Skipped 29 previous similar messages Jun 2 18:07:28 c210 kernel: Lustre: es90-OST000e-osc-ffffa04c7febc800: Connection to es90-OST000e (at 172.16.254.163@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jun 2 18:07:28 c210 kernel: LustreError: 167-0: es90-OST000e-osc-ffffa04c7febc800: This client was evicted by es90-OST000e; in progress operations using this service will fail. Jun 2 18:07:28 c210 kernel: Lustre: es90-OST000e-osc-ffffa04c7febc800: Connection restored to 172.16.254.163@o2ib (at 172.16.254.163@o2ib) Jun 2 18:07:33 c210 systemd-logind: New session 478 of user root. Jun 2 18:07:33 c210 systemd: Started Session 478 of user root. Jun 2 18:07:33 c210 systemd: Starting Session 478 of user root. Jun 2 18:07:33 c210 systemd-logind: Removed session 478. Jun 2 18:07:46 c210 systemd-logind: New session 479 of user root. Jun 2 18:07:46 c210 systemd: Started Session 479 of user root. Jun 2 18:07:46 c210 systemd: Starting Session 479 of user root. Jun 2 18:08:50 c210 kernel: INFO: task jbd2/dm-0-8:536 blocked for more than 120 seconds. Jun 2 18:08:50 c210 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 2 18:08:50 c210 kernel: jbd2/dm-0-8 D ffffa03c3bc5bf40 0 536 2 0x00000000 Jun 2 18:08:50 c210 kernel: Call Trace: Jun 2 18:08:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:08:50 c210 kernel: [] schedule+0x29/0x70 Jun 2 18:08:50 c210 kernel: [] schedule_timeout+0x239/0x2c0 Jun 2 18:08:50 c210 kernel: [] ? __blk_run_queue+0x39/0x50 Jun 2 18:08:50 c210 kernel: [] ? ktime_get_ts64+0x52/0xf0 Jun 2 18:08:50 c210 kernel: [] ? delayacct_end+0x8f/0xb0 Jun 2 18:08:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:08:50 c210 kernel: [] io_schedule_timeout+0xad/0x130 Jun 2 18:08:50 c210 kernel: [] io_schedule+0x18/0x20 Jun 2 18:08:50 c210 kernel: [] bit_wait_io+0x11/0x50 Jun 2 18:08:50 c210 kernel: [] __wait_on_bit+0x67/0x90 Jun 2 18:08:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:08:50 c210 kernel: [] out_of_line_wait_on_bit+0x81/0xb0 Jun 2 18:08:50 c210 kernel: [] ? wake_bit_function+0x40/0x40 Jun 2 18:08:50 c210 kernel: [] __wait_on_buffer+0x2a/0x30 Jun 2 18:08:50 c210 kernel: [] jbd2_journal_commit_transaction+0x1781/0x19b0 [jbd2] Jun 2 18:08:50 c210 kernel: [] kjournald2+0xc9/0x260 [jbd2] Jun 2 18:08:50 c210 kernel: [] ? wake_up_atomic_t+0x30/0x30 Jun 2 18:08:50 c210 kernel: [] ? commit_timeout+0x10/0x10 [jbd2] Jun 2 18:08:50 c210 kernel: [] kthread+0xd1/0xe0 Jun 2 18:08:50 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:08:50 c210 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Jun 2 18:08:50 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:08:50 c210 kernel: INFO: task auditd:816 blocked for more than 120 seconds. Jun 2 18:08:50 c210 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 2 18:08:50 c210 kernel: auditd D ffffa03c3bc5eeb0 0 816 1 0x00000000 Jun 2 18:08:50 c210 kernel: Call Trace: Jun 2 18:08:50 c210 kernel: [] schedule+0x29/0x70 Jun 2 18:08:50 c210 kernel: [] jbd2_log_wait_commit+0xc5/0x140 [jbd2] Jun 2 18:08:50 c210 kernel: [] ? wake_up_atomic_t+0x30/0x30 Jun 2 18:08:50 c210 kernel: [] jbd2_complete_transaction+0x52/0xa0 [jbd2] Jun 2 18:08:50 c210 kernel: [] ext4_sync_file+0x292/0x320 [ext4] Jun 2 18:08:50 c210 kernel: [] do_fsync+0x67/0xb0 Jun 2 18:08:50 c210 kernel: [] SyS_fsync+0x10/0x20 Jun 2 18:08:50 c210 kernel: [] system_call_fastpath+0x1c/0x21 Jun 2 18:10:01 c210 systemd: Started Session 480 of user root. Jun 2 18:10:01 c210 systemd: Starting Session 480 of user root. Jun 2 18:10:50 c210 kernel: INFO: task jbd2/dm-0-8:536 blocked for more than 120 seconds. Jun 2 18:10:50 c210 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 2 18:10:50 c210 kernel: jbd2/dm-0-8 D ffffa03c3bc5bf40 0 536 2 0x00000000 Jun 2 18:10:50 c210 kernel: Call Trace: Jun 2 18:10:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:10:50 c210 kernel: [] schedule+0x29/0x70 Jun 2 18:10:50 c210 kernel: [] schedule_timeout+0x239/0x2c0 Jun 2 18:10:50 c210 kernel: [] ? __blk_run_queue+0x39/0x50 Jun 2 18:10:50 c210 kernel: [] ? ktime_get_ts64+0x52/0xf0 Jun 2 18:10:50 c210 kernel: [] ? delayacct_end+0x8f/0xb0 Jun 2 18:10:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:10:50 c210 kernel: [] io_schedule_timeout+0xad/0x130 Jun 2 18:10:50 c210 kernel: [] io_schedule+0x18/0x20 Jun 2 18:10:50 c210 kernel: [] bit_wait_io+0x11/0x50 Jun 2 18:10:50 c210 kernel: [] __wait_on_bit+0x67/0x90 Jun 2 18:10:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:10:50 c210 kernel: [] out_of_line_wait_on_bit+0x81/0xb0 Jun 2 18:10:50 c210 kernel: [] ? wake_bit_function+0x40/0x40 Jun 2 18:10:50 c210 kernel: [] __wait_on_buffer+0x2a/0x30 Jun 2 18:10:50 c210 kernel: [] jbd2_journal_commit_transaction+0x1781/0x19b0 [jbd2] Jun 2 18:10:50 c210 kernel: [] kjournald2+0xc9/0x260 [jbd2] Jun 2 18:10:50 c210 kernel: [] ? wake_up_atomic_t+0x30/0x30 Jun 2 18:10:50 c210 kernel: [] ? commit_timeout+0x10/0x10 [jbd2] Jun 2 18:10:50 c210 kernel: [] kthread+0xd1/0xe0 Jun 2 18:10:50 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:10:50 c210 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Jun 2 18:10:50 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:10:50 c210 kernel: INFO: task auditd:816 blocked for more than 120 seconds. Jun 2 18:10:50 c210 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 2 18:10:50 c210 kernel: auditd D ffffa03c3bc5eeb0 0 816 1 0x00000000 Jun 2 18:10:50 c210 kernel: Call Trace: Jun 2 18:10:50 c210 kernel: [] schedule+0x29/0x70 Jun 2 18:10:50 c210 kernel: [] jbd2_log_wait_commit+0xc5/0x140 [jbd2] Jun 2 18:10:50 c210 kernel: [] ? wake_up_atomic_t+0x30/0x30 Jun 2 18:10:50 c210 kernel: [] jbd2_complete_transaction+0x52/0xa0 [jbd2] Jun 2 18:10:50 c210 kernel: [] ext4_sync_file+0x292/0x320 [ext4] Jun 2 18:10:50 c210 kernel: [] do_fsync+0x67/0xb0 Jun 2 18:10:50 c210 kernel: [] SyS_fsync+0x10/0x20 Jun 2 18:10:50 c210 kernel: [] system_call_fastpath+0x1c/0x21 Jun 2 18:12:50 c210 kernel: INFO: task jbd2/dm-0-8:536 blocked for more than 120 seconds. Jun 2 18:12:50 c210 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 2 18:12:50 c210 kernel: jbd2/dm-0-8 D ffffa03c3bc5bf40 0 536 2 0x00000000 Jun 2 18:12:50 c210 kernel: Call Trace: Jun 2 18:12:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:12:50 c210 kernel: [] schedule+0x29/0x70 Jun 2 18:12:50 c210 kernel: [] schedule_timeout+0x239/0x2c0 Jun 2 18:12:50 c210 kernel: [] ? __blk_run_queue+0x39/0x50 Jun 2 18:12:50 c210 kernel: [] ? ktime_get_ts64+0x52/0xf0 Jun 2 18:12:50 c210 kernel: [] ? delayacct_end+0x8f/0xb0 Jun 2 18:12:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:12:50 c210 kernel: [] io_schedule_timeout+0xad/0x130 Jun 2 18:12:50 c210 kernel: [] io_schedule+0x18/0x20 Jun 2 18:12:50 c210 kernel: [] bit_wait_io+0x11/0x50 Jun 2 18:12:50 c210 kernel: [] __wait_on_bit+0x67/0x90 Jun 2 18:12:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:12:50 c210 kernel: [] out_of_line_wait_on_bit+0x81/0xb0 Jun 2 18:12:50 c210 kernel: [] ? wake_bit_function+0x40/0x40 Jun 2 18:12:50 c210 kernel: [] __wait_on_buffer+0x2a/0x30 Jun 2 18:12:50 c210 kernel: [] jbd2_journal_commit_transaction+0x1781/0x19b0 [jbd2] Jun 2 18:12:50 c210 kernel: [] kjournald2+0xc9/0x260 [jbd2] Jun 2 18:12:50 c210 kernel: [] ? wake_up_atomic_t+0x30/0x30 Jun 2 18:12:50 c210 kernel: [] ? commit_timeout+0x10/0x10 [jbd2] Jun 2 18:12:50 c210 kernel: [] kthread+0xd1/0xe0 Jun 2 18:12:50 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:12:50 c210 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Jun 2 18:12:50 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:12:50 c210 kernel: INFO: task auditd:816 blocked for more than 120 seconds. Jun 2 18:12:50 c210 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 2 18:12:50 c210 kernel: auditd D ffffa03c3bc5eeb0 0 816 1 0x00000000 Jun 2 18:12:50 c210 kernel: Call Trace: Jun 2 18:12:50 c210 kernel: [] schedule+0x29/0x70 Jun 2 18:12:50 c210 kernel: [] jbd2_log_wait_commit+0xc5/0x140 [jbd2] Jun 2 18:12:50 c210 kernel: [] ? wake_up_atomic_t+0x30/0x30 Jun 2 18:12:50 c210 kernel: [] jbd2_complete_transaction+0x52/0xa0 [jbd2] Jun 2 18:12:50 c210 kernel: [] ext4_sync_file+0x292/0x320 [ext4] Jun 2 18:12:50 c210 kernel: [] do_fsync+0x67/0xb0 Jun 2 18:12:50 c210 kernel: [] SyS_fsync+0x10/0x20 Jun 2 18:12:50 c210 kernel: [] system_call_fastpath+0x1c/0x21 Jun 2 18:14:50 c210 kernel: INFO: task jbd2/dm-0-8:536 blocked for more than 120 seconds. Jun 2 18:14:50 c210 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 2 18:14:50 c210 kernel: jbd2/dm-0-8 D ffffa03c3bc5bf40 0 536 2 0x00000000 Jun 2 18:14:50 c210 kernel: Call Trace: Jun 2 18:14:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:14:50 c210 kernel: [] schedule+0x29/0x70 Jun 2 18:14:50 c210 kernel: [] schedule_timeout+0x239/0x2c0 Jun 2 18:14:50 c210 kernel: [] ? __blk_run_queue+0x39/0x50 Jun 2 18:14:50 c210 kernel: [] ? ktime_get_ts64+0x52/0xf0 Jun 2 18:14:50 c210 kernel: [] ? delayacct_end+0x8f/0xb0 Jun 2 18:14:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:14:50 c210 kernel: [] io_schedule_timeout+0xad/0x130 Jun 2 18:14:50 c210 kernel: [] io_schedule+0x18/0x20 Jun 2 18:14:50 c210 kernel: [] bit_wait_io+0x11/0x50 Jun 2 18:14:50 c210 kernel: [] __wait_on_bit+0x67/0x90 Jun 2 18:14:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:14:50 c210 kernel: [] out_of_line_wait_on_bit+0x81/0xb0 Jun 2 18:14:50 c210 kernel: [] ? wake_bit_function+0x40/0x40 Jun 2 18:14:50 c210 kernel: [] __wait_on_buffer+0x2a/0x30 Jun 2 18:14:50 c210 kernel: [] jbd2_journal_commit_transaction+0x1781/0x19b0 [jbd2] Jun 2 18:14:50 c210 kernel: [] kjournald2+0xc9/0x260 [jbd2] Jun 2 18:14:50 c210 kernel: [] ? wake_up_atomic_t+0x30/0x30 Jun 2 18:14:50 c210 kernel: [] ? commit_timeout+0x10/0x10 [jbd2] Jun 2 18:14:50 c210 kernel: [] kthread+0xd1/0xe0 Jun 2 18:14:50 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:14:50 c210 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Jun 2 18:14:50 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:14:50 c210 kernel: INFO: task auditd:816 blocked for more than 120 seconds. Jun 2 18:14:50 c210 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 2 18:14:50 c210 kernel: auditd D ffffa03c3bc5eeb0 0 816 1 0x00000000 Jun 2 18:14:50 c210 kernel: Call Trace: Jun 2 18:14:50 c210 kernel: [] schedule+0x29/0x70 Jun 2 18:14:50 c210 kernel: [] jbd2_log_wait_commit+0xc5/0x140 [jbd2] Jun 2 18:14:50 c210 kernel: [] ? wake_up_atomic_t+0x30/0x30 Jun 2 18:14:50 c210 kernel: [] jbd2_complete_transaction+0x52/0xa0 [jbd2] Jun 2 18:14:50 c210 kernel: [] ext4_sync_file+0x292/0x320 [ext4] Jun 2 18:14:50 c210 kernel: [] do_fsync+0x67/0xb0 Jun 2 18:14:50 c210 kernel: [] SyS_fsync+0x10/0x20 Jun 2 18:14:50 c210 kernel: [] system_call_fastpath+0x1c/0x21 Jun 2 18:16:50 c210 kernel: INFO: task jbd2/dm-0-8:536 blocked for more than 120 seconds. Jun 2 18:16:50 c210 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 2 18:16:50 c210 kernel: jbd2/dm-0-8 D ffffa03c3bc5bf40 0 536 2 0x00000000 Jun 2 18:16:50 c210 kernel: Call Trace: Jun 2 18:16:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:16:50 c210 kernel: [] schedule+0x29/0x70 Jun 2 18:16:50 c210 kernel: [] schedule_timeout+0x239/0x2c0 Jun 2 18:16:50 c210 kernel: [] ? __blk_run_queue+0x39/0x50 Jun 2 18:16:50 c210 kernel: [] ? ktime_get_ts64+0x52/0xf0 Jun 2 18:16:50 c210 kernel: [] ? delayacct_end+0x8f/0xb0 Jun 2 18:16:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:16:50 c210 kernel: [] io_schedule_timeout+0xad/0x130 Jun 2 18:16:50 c210 kernel: [] io_schedule+0x18/0x20 Jun 2 18:16:50 c210 kernel: [] bit_wait_io+0x11/0x50 Jun 2 18:16:50 c210 kernel: [] __wait_on_bit+0x67/0x90 Jun 2 18:16:50 c210 kernel: [] ? bit_wait+0x50/0x50 Jun 2 18:16:50 c210 kernel: [] out_of_line_wait_on_bit+0x81/0xb0 Jun 2 18:16:50 c210 kernel: [] ? wake_bit_function+0x40/0x40 Jun 2 18:16:50 c210 kernel: [] __wait_on_buffer+0x2a/0x30 Jun 2 18:16:50 c210 kernel: [] jbd2_journal_commit_transaction+0x1781/0x19b0 [jbd2] Jun 2 18:16:50 c210 kernel: [] kjournald2+0xc9/0x260 [jbd2] Jun 2 18:16:50 c210 kernel: [] ? wake_up_atomic_t+0x30/0x30 Jun 2 18:16:50 c210 kernel: [] ? commit_timeout+0x10/0x10 [jbd2] Jun 2 18:16:50 c210 kernel: [] kthread+0xd1/0xe0 Jun 2 18:16:50 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:16:50 c210 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Jun 2 18:16:50 c210 kernel: [] ? insert_kthread_work+0x40/0x40 Jun 2 18:20:01 c210 systemd: Started Session 481 of user root. Jun 2 18:20:01 c210 systemd: Starting Session 481 of user root. Jun 2 18:27:32 c210 kernel: LustreError: 11-0: es90-OST000e-osc-ffffa04c7febc800: operation ost_set_info to node 172.16.254.163@o2ib failed: rc = -107 Jun 2 18:27:32 c210 kernel: Lustre: es90-OST000e-osc-ffffa04c7febc800: Connection to es90-OST000e (at 172.16.254.163@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jun 2 18:27:32 c210 kernel: LustreError: 167-0: es90-OST000e-osc-ffffa04c7febc800: This client was evicted by es90-OST000e; in progress operations using this service will fail. Jun 2 18:27:32 c210 kernel: Lustre: es90-OST000e-osc-ffffa04c7febc800: Connection restored to 172.16.254.163@o2ib (at 172.16.254.163@o2ib) Jun 2 18:30:01 c210 systemd: Started Session 482 of user root. Jun 2 18:30:01 c210 systemd: Starting Session 482 of user root. Jun 2 18:34:26 c210 systemd-logind: Removed session 479. Jun 2 18:34:30 c210 systemd-logind: New session 483 of user root. Jun 2 18:34:30 c210 systemd: Started Session 483 of user root. Jun 2 18:34:30 c210 systemd: Starting Session 483 of user root. Jun 2 18:34:42 c210 systemd-logind: Removed session 483. Jun 2 18:35:09 c210 systemd-logind: New session 484 of user root. Jun 2 18:35:09 c210 systemd: Started Session 484 of user root. Jun 2 18:35:09 c210 systemd: Starting Session 484 of user root. Jun 2 18:35:11 c210 systemd-logind: Removed session 484. Jun 2 18:35:17 c210 systemd-logind: New session 485 of user root. Jun 2 18:35:17 c210 systemd: Started Session 485 of user root. Jun 2 18:35:17 c210 systemd: Starting Session 485 of user root.