Dec 8 08:31:13 atlas-oss3b4 kernel: [1019177.396353] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88072987e000 Dec 8 08:31:13 atlas-oss3b4 kernel: [1019177.520990] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 08:31:13 atlas-oss3b4 kernel: [1019177.603143] LustreError: 11596:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88080b93ac00 x1453425687516402/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386510053 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 08:31:13 atlas-oss3b4 kernel: [1019177.706947] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 08:31:22 atlas-oss4h1 kernel: [1019179.785145] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fe16bc000 Dec 8 08:31:22 atlas-oss4h1 kernel: [1019179.814430] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdbe66000 Dec 8 08:31:23 atlas-oss4h1 kernel: [1019179.853597] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fe16bc000 Dec 8 08:31:23 atlas-oss4h1 kernel: [1019179.884464] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdbe66000 Dec 8 08:31:23 atlas-oss4h1 kernel: [1019179.884496] LustreError: 113675:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff8810007eb000 x1453430541316010/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386510058 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 08:31:23 atlas-oss4h1 kernel: [1019179.884604] Lustre: atlas2-OST03e0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 08:33:08 atlas-oss4h8 kernel: [1019294.528627] LustreError: 137-5: atlas2-OST0083_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 08:35:13 atlas-oss4h4 kernel: [432955.505683] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc1ebc000 Dec 8 08:35:13 atlas-oss4h4 kernel: [432955.530858] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fb89c0000 Dec 8 08:35:13 atlas-oss4h4 kernel: [432955.569961] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc1ebc000 Dec 8 08:35:13 atlas-oss4h4 kernel: [432955.600798] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fb89c0000 Dec 8 08:35:14 atlas-oss4h4 kernel: [432955.600839] LustreError: 41411:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fe7747c00 x1453430541319048/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386510008 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 08:35:14 atlas-oss4h4 kernel: [432955.600947] Lustre: atlas2-OST01a3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 08:36:13 atlas-oss3b4 kernel: [1019477.582290] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88071aaf0000 Dec 8 08:38:18 atlas-oss4g6 kernel: [1019609.647776] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc97c6000 Dec 8 08:38:18 atlas-oss4g6 kernel: [1019609.677322] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc8f34000 Dec 8 08:38:18 atlas-oss4g6 kernel: [1019609.716430] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc97c6000 Dec 8 08:38:18 atlas-oss4g6 kernel: [1019609.747371] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc8f34000 Dec 8 08:38:19 atlas-oss4g8 kernel: [1019605.298152] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbf212000 Dec 8 08:38:19 atlas-oss4g8 kernel: [1019605.321588] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbf212000 Dec 8 08:38:19 atlas-oss4g8 kernel: [1019605.360608] LustreError: 119713:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ffa143800 x1453430541329925/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386510191 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 08:38:19 atlas-oss4g8 kernel: [1019605.480050] Lustre: atlas2-OST03df: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 08:38:19 atlas-oss4h4 kernel: [433141.232227] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881052858000 Dec 8 08:38:19 atlas-oss4h4 kernel: [433141.258571] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881052858000 Dec 8 08:38:20 atlas-oss4g7 kernel: [1019616.166831] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881022396000 Dec 8 08:38:20 atlas-oss4g7 kernel: [1019616.196888] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881022396000 Dec 8 08:38:20 atlas-oss4h1 kernel: [1019596.590090] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcfce6000 Dec 8 08:38:20 atlas-oss4h1 kernel: [1019596.619821] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcbf9a000 Dec 8 08:38:20 atlas-oss4g7 kernel: [1019616.236322] LustreError: 114081:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff88101c7b0c00 x1453430541329926/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386510191 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 08:38:20 atlas-oss4h1 kernel: [1019596.659363] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcfce6000 Dec 8 08:38:20 atlas-oss4h1 kernel: [1019596.690208] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcbf9a000 Dec 8 08:38:20 atlas-oss4g7 kernel: [1019616.355644] Lustre: atlas2-OST03de: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 08:39:49 atlas-oss4h8 kernel: [1019694.660903] LustreError: 137-5: atlas2-OST0113_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 08:40:50 atlas-oss4h1 kernel: [1019745.603249] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881038030000 Dec 8 08:40:50 atlas-oss4h1 kernel: [1019745.626712] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8810382d2000 Dec 8 08:40:50 atlas-oss4h1 kernel: [1019745.665921] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881038030000 Dec 8 08:40:50 atlas-oss4h1 kernel: [1019745.696388] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff8810382d2000 Dec 8 08:41:14 atlas-oss3b4 kernel: [1019777.768230] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88075b8c0000 Dec 8 08:41:14 atlas-oss3b4 kernel: [1019777.893649] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 08:41:14 atlas-oss3b4 kernel: [1019777.955941] LustreError: 11717:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880760993800 x1453425687538813/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386510654 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 08:41:15 atlas-oss3b4 kernel: [1019778.058384] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 08:43:53 atlas-oss4h4 kernel: [433473.812334] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc8c70000 Dec 8 08:43:53 atlas-oss4h4 kernel: [433473.838546] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc988c000 Dec 8 08:43:53 atlas-oss4h4 kernel: [433473.877865] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc8c70000 Dec 8 08:43:53 atlas-oss4h4 kernel: [433473.908711] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc988c000 Dec 8 08:46:15 atlas-oss3b4 kernel: [1020077.954175] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88072b0ac000 Dec 8 08:47:42 atlas-oss4h4 kernel: [433702.731109] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fb8518000 Dec 8 08:47:42 atlas-oss4h4 kernel: [433702.762917] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fb8518000 Dec 8 08:47:42 atlas-oss4h4 kernel: [433702.793541] LustreError: 41407:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fedaec800 x1453430541370394/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386511041 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 08:47:42 atlas-oss4h4 kernel: [433702.912704] Lustre: atlas2-OST03e3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 08:47:43 atlas-oss4g7 kernel: [1020177.665713] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc7394000 Dec 8 08:47:43 atlas-oss4g7 kernel: [1020177.694670] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc7394000 Dec 8 08:47:43 atlas-oss4h1 kernel: [1020158.088972] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcf4fa000 Dec 8 08:47:43 atlas-oss4h1 kernel: [1020158.120836] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc990e000 Dec 8 08:47:43 atlas-oss4h1 kernel: [1020158.151656] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcf4fa000 Dec 8 08:47:43 atlas-oss4h1 kernel: [1020158.191169] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc990e000 Dec 8 08:47:43 atlas-oss4h1 kernel: [1020158.191204] LustreError: 48680:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff53a8000 x1453430541370396/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386510758 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 08:47:43 atlas-oss4g6 kernel: [1020173.306282] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc86f8000 Dec 8 08:47:43 atlas-oss4g6 kernel: [1020173.330994] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8810514b6000 Dec 8 08:47:43 atlas-oss4h1 kernel: [1020158.191306] Lustre: atlas2-OST01a0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 08:47:43 atlas-oss4g6 kernel: [1020173.370170] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc86f8000 Dec 8 08:47:43 atlas-oss4g6 kernel: [1020173.401183] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff8810514b6000 Dec 8 08:47:43 atlas-oss4g6 kernel: [1020173.401216] LustreError: 118604:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff8df9c00 x1453430541370392/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386510758 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 08:47:43 atlas-oss4g6 kernel: [1020173.401357] Lustre: atlas2-OST03dd: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 08:47:44 atlas-oss4g8 kernel: [1020168.956642] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbf632000 Dec 8 08:47:44 atlas-oss4g8 kernel: [1020168.980153] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbf632000 Dec 8 08:50:14 atlas-oss4h1 kernel: [1020309.261744] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc990e000 Dec 8 08:50:14 atlas-oss4h1 kernel: [1020309.293265] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881039bdc000 Dec 8 08:50:14 atlas-oss4h1 kernel: [1020309.332536] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc990e000 Dec 8 08:50:14 atlas-oss4h1 kernel: [1020309.363191] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881039bdc000 Dec 8 08:51:16 atlas-oss3b4 kernel: [1020378.140135] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88072b338000 Dec 8 08:51:16 atlas-oss3b4 kernel: [1020378.263542] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 08:51:16 atlas-oss3b4 kernel: [1020378.334683] LustreError: 11717:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8808711f8400 x1453425687561971/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386511256 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 08:51:16 atlas-oss3b4 kernel: [1020378.447585] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 08:53:20 atlas-oss4h4 kernel: [434039.630452] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc48e6000 Dec 8 08:53:20 atlas-oss4h4 kernel: [434039.653695] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc0fe4000 Dec 8 08:53:20 atlas-oss4h4 kernel: [434039.692697] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc48e6000 Dec 8 08:53:20 atlas-oss4h4 kernel: [434039.723609] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc0fe4000 Dec 8 08:56:17 atlas-oss3b4 kernel: [1020678.326056] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880729560000 Dec 8 08:56:18 atlas-oss4g7 kernel: [1020691.653159] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc4504000 Dec 8 08:56:18 atlas-oss4g7 kernel: [1020691.685050] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc4504000 Dec 8 08:56:18 atlas-oss4h1 kernel: [1020672.076408] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fce6f8000 Dec 8 08:56:18 atlas-oss4h1 kernel: [1020672.105553] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881001dde000 Dec 8 08:56:18 atlas-oss4g7 kernel: [1020691.715732] LustreError: 114210:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff0464800 x1453430541394623/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386511557 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 08:56:18 atlas-oss4h1 kernel: [1020672.144914] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fce6f8000 Dec 8 08:56:18 atlas-oss4h1 kernel: [1020672.175704] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881001dde000 Dec 8 08:56:18 atlas-oss4g7 kernel: [1020691.835382] Lustre: atlas2-OST03de: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 08:56:18 atlas-oss4g6 kernel: [1020687.293701] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcab02000 Dec 8 08:56:18 atlas-oss4g6 kernel: [1020687.321966] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc8636000 Dec 8 08:56:18 atlas-oss4g6 kernel: [1020687.361309] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcab02000 Dec 8 08:56:18 atlas-oss4g6 kernel: [1020687.392182] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc8636000 Dec 8 08:56:19 atlas-oss4g8 kernel: [1020682.944087] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc71e8000 Dec 8 08:56:19 atlas-oss4g8 kernel: [1020682.967404] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc71e8000 Dec 8 08:56:19 atlas-oss4g8 kernel: [1020683.006637] LustreError: 119630:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff8886800 x1453430541394625/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386511557 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 08:56:19 atlas-oss4g8 kernel: [1020683.125980] Lustre: atlas2-OST03df: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 08:56:22 atlas-oss4h4 kernel: [434221.037766] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbb97e000 Dec 8 08:56:22 atlas-oss4h4 kernel: [434221.060983] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbb97e000 Dec 8 08:58:54 atlas-oss4h1 kernel: [1020827.568412] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdeb2c000 Dec 8 08:58:54 atlas-oss4h1 kernel: [1020827.598070] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881037e82000 Dec 8 08:58:54 atlas-oss4h1 kernel: [1020827.637240] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdeb2c000 Dec 8 08:58:54 atlas-oss4h1 kernel: [1020827.668206] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881037e82000 Dec 8 08:58:54 atlas-oss4h1 kernel: [1020827.668238] LustreError: 42253:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff8810007ffc00 x1453430541397664/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386511708 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 08:58:54 atlas-oss4h1 kernel: [1020827.668353] Lustre: atlas2-OST03e0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:00:38 atlas-oss4g2 kernel: [1020942.071291] LustreError: 137-5: atlas2-OST007d_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 09:00:38 atlas-oss4g3 kernel: [1020941.097531] LustreError: 137-5: atlas2-OST007e_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 09:00:38 atlas-oss4g4 kernel: [1020943.244236] LustreError: 137-5: atlas2-OST007f_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 09:01:18 atlas-oss3b4 kernel: [1020978.512004] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88072fbd4000 Dec 8 09:01:18 atlas-oss3b4 kernel: [1020978.635772] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 09:01:18 atlas-oss3b4 kernel: [1020978.691455] LustreError: 11717:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88076616bc00 x1453425687585146/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386511858 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:01:18 atlas-oss3b4 kernel: [1020978.798708] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 09:02:42 atlas-oss4h4 kernel: [434601.129322] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbbc5e000 Dec 8 09:02:42 atlas-oss4h4 kernel: [434601.158111] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbb6d6000 Dec 8 09:02:43 atlas-oss4h4 kernel: [434601.197061] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbbc5e000 Dec 8 09:02:43 atlas-oss4h4 kernel: [434601.227780] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbb6d6000 Dec 8 09:02:43 atlas-oss4h4 kernel: [434601.227809] LustreError: 41407:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ffe11dc00 x1453430541400702/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386511941 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:02:43 atlas-oss4h4 kernel: [434601.227928] Lustre: atlas2-OST01a3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:05:45 atlas-oss4h1 kernel: [1021237.894515] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd5536000 Dec 8 09:05:45 atlas-oss4h1 kernel: [1021237.917834] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdf28c000 Dec 8 09:05:45 atlas-oss4h1 kernel: [1021237.957389] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd5536000 Dec 8 09:05:45 atlas-oss4h1 kernel: [1021237.996367] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdf28c000 Dec 8 09:05:49 atlas-oss4h4 kernel: [434786.855871] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881001f4e000 Dec 8 09:05:49 atlas-oss4h4 kernel: [434786.885873] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881001f4e000 Dec 8 09:05:49 atlas-oss4g7 kernel: [1021261.790481] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbc634000 Dec 8 09:05:49 atlas-oss4g7 kernel: [1021261.814549] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbc634000 Dec 8 09:05:50 atlas-oss4g6 kernel: [1021257.431030] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fccebc000 Dec 8 09:05:50 atlas-oss4g6 kernel: [1021257.461630] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fe4a10000 Dec 8 09:05:50 atlas-oss4g6 kernel: [1021257.501146] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fccebc000 Dec 8 09:05:50 atlas-oss4g6 kernel: [1021257.532079] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fe4a10000 Dec 8 09:05:50 atlas-oss4g6 kernel: [1021257.532108] LustreError: 118531:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff8af8400 x1453430541411835/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386512124 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:05:50 atlas-oss4g6 kernel: [1021257.532205] Lustre: atlas2-OST03dd: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:05:50 atlas-oss4g8 kernel: [1021253.081413] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbfcec000 Dec 8 09:05:50 atlas-oss4g8 kernel: [1021253.111644] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbfcec000 Dec 8 09:06:18 atlas-oss3b4 kernel: [1021278.697936] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880761db6000 Dec 8 09:08:21 atlas-oss4h1 kernel: [1021393.386497] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdf28c000 Dec 8 09:08:21 atlas-oss4h1 kernel: [1021393.410017] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdd274000 Dec 8 09:08:21 atlas-oss4h1 kernel: [1021393.449170] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdf28c000 Dec 8 09:08:21 atlas-oss4h1 kernel: [1021393.480163] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdd274000 Dec 8 09:10:06 atlas-oss4h5 kernel: [1021505.148417] LustreError: 137-5: atlas2-OST02c0_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 09:10:06 atlas-oss4h5 kernel: [1021505.148420] LustreError: 137-5: atlas2-OST0350_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 09:11:19 atlas-oss3b4 kernel: [1021578.883874] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8807317ec000 Dec 8 09:11:19 atlas-oss3b4 kernel: [1021579.010034] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 09:11:19 atlas-oss3b4 kernel: [1021579.079167] LustreError: 11709:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88075d8d1c00 x1453425687606810/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386512459 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:11:20 atlas-oss3b4 kernel: [1021579.188116] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 09:11:22 atlas-oss4h4 kernel: [435119.435979] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbcd2c000 Dec 8 09:11:22 atlas-oss4h4 kernel: [435119.465829] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbb1f8000 Dec 8 09:11:22 atlas-oss4h4 kernel: [435119.505182] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbcd2c000 Dec 8 09:11:22 atlas-oss4h4 kernel: [435119.536067] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbb1f8000 Dec 8 09:15:12 atlas-oss4g7 kernel: [1021823.289363] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc8e22000 Dec 8 09:15:12 atlas-oss4g7 kernel: [1021823.312712] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc8e22000 Dec 8 09:15:12 atlas-oss4g7 kernel: [1021823.351902] LustreError: 113999:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fe8952400 x1453430541421975/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386512408 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:15:12 atlas-oss4h1 kernel: [1021803.712620] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fda270000 Dec 8 09:15:12 atlas-oss4h1 kernel: [1021803.739421] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdcc92000 Dec 8 09:15:12 atlas-oss4h1 kernel: [1021803.778767] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fda270000 Dec 8 09:15:12 atlas-oss4g7 kernel: [1021823.462544] Lustre: atlas2-OST03de: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:15:12 atlas-oss4h1 kernel: [1021803.809598] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdcc92000 Dec 8 09:15:12 atlas-oss4h1 kernel: [1021803.809627] LustreError: 113676:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff6a0e400 x1453430541421971/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386512408 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:15:12 atlas-oss4g6 kernel: [1021818.929911] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc8f6c000 Dec 8 09:15:12 atlas-oss4g6 kernel: [1021818.960066] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8810264d8000 Dec 8 09:15:12 atlas-oss4h1 kernel: [1021803.809726] Lustre: atlas2-OST01a0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:15:12 atlas-oss4g6 kernel: [1021818.999156] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc8f6c000 Dec 8 09:15:12 atlas-oss4g6 kernel: [1021819.030061] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff8810264d8000 Dec 8 09:15:13 atlas-oss4g8 kernel: [1021814.580289] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcbffe000 Dec 8 09:15:13 atlas-oss4g8 kernel: [1021814.605503] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcbffe000 Dec 8 09:15:13 atlas-oss4g8 kernel: [1021814.644665] LustreError: 119851:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff88102ea31c00 x1453430541421976/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386512691 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:15:13 atlas-oss4g8 kernel: [1021814.755393] Lustre: atlas2-OST03df: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:15:13 atlas-oss4h4 kernel: [435350.514363] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fa67ea000 Dec 8 09:15:13 atlas-oss4h4 kernel: [435350.545750] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fa67ea000 Dec 8 09:15:14 atlas-oss4h4 kernel: [435350.584962] LustreError: 61445:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fe1d8ac00 x1453430541421972/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386512691 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:15:14 atlas-oss4h4 kernel: [435350.695887] Lustre: atlas2-OST03e3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:16:20 atlas-oss3b4 kernel: [1021879.069826] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8807629b8000 Dec 8 09:17:48 atlas-oss4h1 kernel: [1021959.204616] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcc57a000 Dec 8 09:17:48 atlas-oss4h1 kernel: [1021959.231977] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fda270000 Dec 8 09:17:48 atlas-oss4h1 kernel: [1021959.271270] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcc57a000 Dec 8 09:17:48 atlas-oss4h1 kernel: [1021959.302160] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fda270000 Dec 8 09:20:49 atlas-oss4h4 kernel: [435685.254083] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbfb6c000 Dec 8 09:20:49 atlas-oss4h4 kernel: [435685.281054] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc89e8000 Dec 8 09:20:49 atlas-oss4h4 kernel: [435685.320230] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbfb6c000 Dec 8 09:20:49 atlas-oss4h4 kernel: [435685.351211] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc89e8000 Dec 8 09:21:21 atlas-oss3b4 kernel: [1022179.255815] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88071ad90000 Dec 8 09:21:21 atlas-oss3b4 kernel: [1022179.381198] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 09:21:21 atlas-oss3b4 kernel: [1022179.450890] LustreError: 11709:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88076a36c000 x1453425687627617/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386513061 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:21:21 atlas-oss3b4 kernel: [1022179.558788] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 09:23:47 atlas-oss4h1 kernel: [1022317.700058] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdc82a000 Dec 8 09:23:47 atlas-oss4h1 kernel: [1022317.723852] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcd254000 Dec 8 09:23:47 atlas-oss4h1 kernel: [1022317.762825] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdc82a000 Dec 8 09:23:47 atlas-oss4h1 kernel: [1022317.793700] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcd254000 Dec 8 09:23:47 atlas-oss4g6 kernel: [1022332.917348] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fe3a86000 Dec 8 09:23:47 atlas-oss4g6 kernel: [1022332.941222] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdf41a000 Dec 8 09:23:48 atlas-oss4g6 kernel: [1022332.980334] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fe3a86000 Dec 8 09:23:48 atlas-oss4g6 kernel: [1022333.011420] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdf41a000 Dec 8 09:23:48 atlas-oss4g6 kernel: [1022333.011450] LustreError: 118643:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff2b87400 x1453430541436172/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386513207 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:23:48 atlas-oss4g6 kernel: [1022333.011562] Lustre: atlas2-OST03dd: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:23:51 atlas-oss4g7 kernel: [1022341.596018] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcb9f2000 Dec 8 09:23:51 atlas-oss4g7 kernel: [1022341.623992] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcb9f2000 Dec 8 09:23:52 atlas-oss4g8 kernel: [1022332.886955] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88104eae2000 Dec 8 09:23:52 atlas-oss4g8 kernel: [1022332.913072] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff88104eae2000 Dec 8 09:23:53 atlas-oss4h4 kernel: [435868.821028] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc89e8000 Dec 8 09:23:53 atlas-oss4h4 kernel: [435868.852703] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc89e8000 Dec 8 09:26:22 atlas-oss3b4 kernel: [1022479.441702] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88071b590000 Dec 8 09:27:15 atlas-oss4h1 kernel: [1022525.022728] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd23f4000 Dec 8 09:27:15 atlas-oss4h1 kernel: [1022525.053729] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd0ebe000 Dec 8 09:27:15 atlas-oss4h1 kernel: [1022525.092837] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd23f4000 Dec 8 09:27:15 atlas-oss4h1 kernel: [1022525.123730] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd0ebe000 Dec 8 09:27:15 atlas-oss4h1 kernel: [1022525.123761] LustreError: 113676:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff253d000 x1453430541439222/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386513410 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:27:15 atlas-oss4h1 kernel: [1022525.123900] Lustre: atlas2-OST03e0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:30:14 atlas-oss4h4 kernel: [436248.912575] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fb9376000 Dec 8 09:30:14 atlas-oss4h4 kernel: [436248.940785] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc89e8000 Dec 8 09:30:14 atlas-oss4h4 kernel: [436248.979769] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fb9376000 Dec 8 09:30:14 atlas-oss4h4 kernel: [436249.010672] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc89e8000 Dec 8 09:30:14 atlas-oss4h4 kernel: [436249.010703] LustreError: 42111:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fead42800 x1453430541442260/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386513308 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:30:14 atlas-oss4h4 kernel: [436249.010831] Lustre: atlas2-OST01a3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:31:23 atlas-oss3b4 kernel: [1022779.627643] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880731a10000 Dec 8 09:31:23 atlas-oss3b4 kernel: [1022779.752183] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 09:31:23 atlas-oss3b4 kernel: [1022779.822632] LustreError: 11718:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8807752c8000 x1453425687648412/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386513663 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:31:23 atlas-oss3b4 kernel: [1022779.928735] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 09:33:16 atlas-oss4h1 kernel: [1022885.677772] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880ff592c000 Dec 8 09:33:16 atlas-oss4h1 kernel: [1022885.701130] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881038bf0000 Dec 8 09:33:16 atlas-oss4h1 kernel: [1022885.740465] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880ff592c000 Dec 8 09:33:16 atlas-oss4h1 kernel: [1022885.771638] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881038bf0000 Dec 8 09:33:18 atlas-oss4g7 kernel: [1022907.414129] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc3466000 Dec 8 09:33:18 atlas-oss4g7 kernel: [1022907.442747] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc3466000 Dec 8 09:33:19 atlas-oss4g7 kernel: [1022907.481793] LustreError: 22016:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff0abd400 x1453430541453164/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386513491 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:33:19 atlas-oss4g7 kernel: [1022907.592507] Lustre: atlas2-OST03de: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:33:19 atlas-oss4g6 kernel: [1022903.054678] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc8744000 Dec 8 09:33:19 atlas-oss4g6 kernel: [1022903.080930] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fca388000 Dec 8 09:33:19 atlas-oss4g6 kernel: [1022903.120347] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc8744000 Dec 8 09:33:19 atlas-oss4g6 kernel: [1022903.151193] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fca388000 Dec 8 09:33:19 atlas-oss4g8 kernel: [1022898.705058] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fe73dc000 Dec 8 09:33:19 atlas-oss4g8 kernel: [1022898.736777] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fe73dc000 Dec 8 09:33:19 atlas-oss4g8 kernel: [1022898.768010] LustreError: 121265:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff881006b1c000 x1453430541453163/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386513774 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:33:20 atlas-oss4g8 kernel: [1022898.887513] Lustre: atlas2-OST03df: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:33:20 atlas-oss4h4 kernel: [436434.639110] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbb566000 Dec 8 09:33:20 atlas-oss4h4 kernel: [436434.668488] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbb566000 Dec 8 09:36:23 atlas-oss3b4 kernel: [1023079.813583] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88071aa36000 Dec 8 09:36:42 atlas-oss4h1 kernel: [1023090.840824] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd9416000 Dec 8 09:36:42 atlas-oss4h1 kernel: [1023090.865984] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd3c5a000 Dec 8 09:36:42 atlas-oss4h1 kernel: [1023090.904972] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd9416000 Dec 8 09:36:42 atlas-oss4h1 kernel: [1023090.946479] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd3c5a000 Dec 8 09:37:36 atlas-oss4h5 kernel: [1023151.626203] LustreError: 137-5: atlas2-OST0230_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 09:38:53 atlas-oss4h4 kernel: [436767.219236] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fa5eb0000 Dec 8 09:38:53 atlas-oss4h4 kernel: [436767.248518] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fa2708000 Dec 8 09:38:53 atlas-oss4h4 kernel: [436767.287618] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fa5eb0000 Dec 8 09:38:53 atlas-oss4h4 kernel: [436767.318503] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fa2708000 Dec 8 09:41:24 atlas-oss3b4 kernel: [1023379.999526] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88072a576000 Dec 8 09:41:24 atlas-oss3b4 kernel: [1023380.125878] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 09:41:25 atlas-oss3b4 kernel: [1023380.194338] LustreError: 11718:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880764ff6800 x1453425687669041/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386514264 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:41:25 atlas-oss3b4 kernel: [1023380.299074] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 09:42:42 atlas-oss4g8 kernel: [1023460.203940] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc9eaa000 Dec 8 09:42:42 atlas-oss4g8 kernel: [1023460.231527] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc9eaa000 Dec 8 09:42:43 atlas-oss4h4 kernel: [436996.138013] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc1552000 Dec 8 09:42:43 atlas-oss4h4 kernel: [436996.163263] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc1552000 Dec 8 09:42:43 atlas-oss4h4 kernel: [436996.202289] LustreError: 42111:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fe2304c00 x1453430541463302/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386514058 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:42:43 atlas-oss4h4 kernel: [436996.313206] Lustre: atlas2-OST03e3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:42:43 atlas-oss4g7 kernel: [1023471.072618] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc551c000 Dec 8 09:42:43 atlas-oss4g7 kernel: [1023471.096076] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc551c000 Dec 8 09:42:43 atlas-oss4h1 kernel: [1023451.495874] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd0ebe000 Dec 8 09:42:43 atlas-oss4h1 kernel: [1023451.523086] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcbd7c000 Dec 8 09:42:44 atlas-oss4h1 kernel: [1023451.562399] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd0ebe000 Dec 8 09:42:44 atlas-oss4h1 kernel: [1023451.593285] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcbd7c000 Dec 8 09:42:44 atlas-oss4g6 kernel: [1023466.713168] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881069e70000 Dec 8 09:42:44 atlas-oss4g6 kernel: [1023466.744555] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdd0ba000 Dec 8 09:42:44 atlas-oss4h1 kernel: [1023451.593317] LustreError: 113527:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff881009679800 x1453430541463296/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386514341 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:42:44 atlas-oss4h1 kernel: [1023451.593434] Lustre: atlas2-OST01a0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:42:44 atlas-oss4g6 kernel: [1023466.784056] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881069e70000 Dec 8 09:42:44 atlas-oss4g6 kernel: [1023466.815051] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdd0ba000 Dec 8 09:42:44 atlas-oss4g6 kernel: [1023466.815080] LustreError: 118630:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff88106d88d400 x1453430541463298/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386514341 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:42:44 atlas-oss4g6 kernel: [1023466.815198] Lustre: atlas2-OST03dd: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:46:09 atlas-oss4h1 kernel: [1023656.658929] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcdeb8000 Dec 8 09:46:09 atlas-oss4h1 kernel: [1023656.687624] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd3ca0000 Dec 8 09:46:09 atlas-oss4h1 kernel: [1023656.726906] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcdeb8000 Dec 8 09:46:09 atlas-oss4h1 kernel: [1023656.757798] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd3ca0000 Dec 8 09:46:25 atlas-oss3b4 kernel: [1023680.185460] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880729de8000 Dec 8 09:48:20 atlas-oss4h4 kernel: [437333.037346] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881051aaa000 Dec 8 09:48:20 atlas-oss4h4 kernel: [437333.063520] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbf15c000 Dec 8 09:48:20 atlas-oss4h4 kernel: [437333.102922] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881051aaa000 Dec 8 09:48:20 atlas-oss4h4 kernel: [437333.134012] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbf15c000 Dec 8 09:51:17 atlas-oss4g8 kernel: [1023974.191375] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcbb62000 Dec 8 09:51:17 atlas-oss4g8 kernel: [1023974.218133] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcbb62000 Dec 8 09:51:17 atlas-oss4g8 kernel: [1023974.257337] LustreError: 119822:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ffe6b1c00 x1453430541477497/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386514857 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:51:17 atlas-oss4g8 kernel: [1023974.368581] Lustre: atlas2-OST03df: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:51:19 atlas-oss4h1 kernel: [1023965.483313] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880ffc33a000 Dec 8 09:51:19 atlas-oss4h1 kernel: [1023965.507484] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd3e0c000 Dec 8 09:51:19 atlas-oss4h1 kernel: [1023965.546631] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880ffc33a000 Dec 8 09:51:19 atlas-oss4h1 kernel: [1023965.577444] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd3e0c000 Dec 8 09:51:22 atlas-oss4h4 kernel: [437514.444670] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbf434000 Dec 8 09:51:22 atlas-oss4h4 kernel: [437514.470968] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbf434000 Dec 8 09:51:23 atlas-oss4g7 kernel: [1023989.379279] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fb9d48000 Dec 8 09:51:23 atlas-oss4g7 kernel: [1023989.406963] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fb9d48000 Dec 8 09:51:23 atlas-oss4g7 kernel: [1023989.446338] LustreError: 115249:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff1980000 x1453430541477503/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386514857 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:51:23 atlas-oss4g7 kernel: [1023989.556925] Lustre: atlas2-OST03de: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:51:23 atlas-oss4g6 kernel: [1023985.019829] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fce264000 Dec 8 09:51:23 atlas-oss4g6 kernel: [1023985.046395] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fca388000 Dec 8 09:51:23 atlas-oss4g6 kernel: [1023985.085774] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fce264000 Dec 8 09:51:23 atlas-oss4g6 kernel: [1023985.116405] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fca388000 Dec 8 09:51:26 atlas-oss3b4 kernel: [1023980.371406] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88072a736000 Dec 8 09:51:26 atlas-oss3b4 kernel: [1023980.495466] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 09:51:26 atlas-oss3b4 kernel: [1023980.567054] LustreError: 11576:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88076ab48400 x1453425687689764/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386514866 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:51:26 atlas-oss3b4 kernel: [1023980.670027] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 09:55:36 atlas-oss4h1 kernel: [1024222.477031] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd7434000 Dec 8 09:55:36 atlas-oss4h1 kernel: [1024222.500374] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd03de000 Dec 8 09:55:36 atlas-oss4h1 kernel: [1024222.539631] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd7434000 Dec 8 09:55:36 atlas-oss4h1 kernel: [1024222.579041] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd03de000 Dec 8 09:55:36 atlas-oss4h1 kernel: [1024222.579080] LustreError: 113675:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff88100aa9b800 x1453430541483580/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386514828 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:55:36 atlas-oss4h1 kernel: [1024222.579195] Lustre: atlas2-OST03e0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 09:55:38 atlas-oss4g4 kernel: [1024236.141137] LustreError: 137-5: atlas2-OST007f_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 09:56:27 atlas-oss3b4 kernel: [1024280.557356] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88075d2f4000 Dec 8 09:57:43 atlas-oss4h4 kernel: [437894.536226] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc1648000 Dec 8 09:57:43 atlas-oss4h4 kernel: [437894.567760] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fb9cfa000 Dec 8 09:57:43 atlas-oss4h4 kernel: [437894.598639] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc1648000 Dec 8 09:57:43 atlas-oss4h4 kernel: [437894.637830] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fb9cfa000 Dec 8 09:57:43 atlas-oss4h4 kernel: [437894.637861] LustreError: 61461:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fe2932c00 x1453430541483585/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386514958 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 09:57:43 atlas-oss4h4 kernel: [437894.637983] Lustre: atlas2-OST01a3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:00:46 atlas-oss4h1 kernel: [1024531.301417] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881003656000 Dec 8 10:00:46 atlas-oss4h1 kernel: [1024531.329442] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd07b2000 Dec 8 10:00:46 atlas-oss4h1 kernel: [1024531.368798] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881003656000 Dec 8 10:00:46 atlas-oss4h1 kernel: [1024531.399683] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd07b2000 Dec 8 10:00:49 atlas-oss4g8 kernel: [1024544.328702] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd1b50000 Dec 8 10:00:49 atlas-oss4g8 kernel: [1024544.353302] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd1b50000 Dec 8 10:00:49 atlas-oss4h4 kernel: [438080.262777] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc0dfa000 Dec 8 10:00:49 atlas-oss4h4 kernel: [438080.285848] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc0dfa000 Dec 8 10:00:50 atlas-oss4g7 kernel: [1024555.197381] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd11dc000 Dec 8 10:00:50 atlas-oss4g7 kernel: [1024555.225779] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd11dc000 Dec 8 10:00:50 atlas-oss4g6 kernel: [1024550.837931] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc8a56000 Dec 8 10:00:50 atlas-oss4g6 kernel: [1024550.865335] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd0fe4000 Dec 8 10:00:50 atlas-oss4g6 kernel: [1024550.904660] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc8a56000 Dec 8 10:00:50 atlas-oss4g6 kernel: [1024550.935507] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd0fe4000 Dec 8 10:00:50 atlas-oss4g6 kernel: [1024550.935537] LustreError: 118578:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff4ff4400 x1453430541494604/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386515424 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:00:50 atlas-oss4g6 kernel: [1024550.935655] Lustre: atlas2-OST03dd: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:01:28 atlas-oss3b4 kernel: [1024580.743287] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880718d34000 Dec 8 10:01:28 atlas-oss3b4 kernel: [1024580.867120] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 10:01:28 atlas-oss3b4 kernel: [1024580.938774] LustreError: 11604:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88075fc5e400 x1453425687710452/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386515468 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:01:28 atlas-oss3b4 kernel: [1024581.049910] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 10:05:03 atlas-oss4h1 kernel: [1024788.295135] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcace0000 Dec 8 10:05:03 atlas-oss4h1 kernel: [1024788.321656] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fe0852000 Dec 8 10:05:03 atlas-oss4h1 kernel: [1024788.360781] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcace0000 Dec 8 10:05:03 atlas-oss4h1 kernel: [1024788.391846] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fe0852000 Dec 8 10:05:05 atlas-oss4h5 kernel: [1024797.066240] LustreError: 137-5: atlas2-OST02c0_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 10:06:22 atlas-oss4h4 kernel: [438412.842884] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fb9de6000 Dec 8 10:06:22 atlas-oss4h4 kernel: [438412.866335] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fb8568000 Dec 8 10:06:22 atlas-oss4h4 kernel: [438412.905532] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fb9de6000 Dec 8 10:06:22 atlas-oss4h4 kernel: [438412.936265] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fb8568000 Dec 8 10:06:28 atlas-oss3b4 kernel: [1024880.929223] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880765e44000 Dec 8 10:10:12 atlas-oss4g7 kernel: [1025116.696268] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcc224000 Dec 8 10:10:12 atlas-oss4g7 kernel: [1025116.723735] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcc224000 Dec 8 10:10:13 atlas-oss4g7 kernel: [1025116.762740] LustreError: 115244:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ffeb9dc00 x1453430541504745/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386515708 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:10:13 atlas-oss4h1 kernel: [1025097.119524] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdd2d4000 Dec 8 10:10:13 atlas-oss4h1 kernel: [1025097.151338] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd9d54000 Dec 8 10:10:13 atlas-oss4g7 kernel: [1025116.873693] Lustre: atlas2-OST03de: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:10:13 atlas-oss4h1 kernel: [1025097.190376] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdd2d4000 Dec 8 10:10:13 atlas-oss4h1 kernel: [1025097.221346] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd9d54000 Dec 8 10:10:13 atlas-oss4h1 kernel: [1025097.221375] LustreError: 42231:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff24a6400 x1453430541504740/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386515708 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:10:13 atlas-oss4g6 kernel: [1025112.336817] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc7d40000 Dec 8 10:10:13 atlas-oss4g6 kernel: [1025112.364000] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcce2c000 Dec 8 10:10:13 atlas-oss4g6 kernel: [1025112.403084] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc7d40000 Dec 8 10:10:13 atlas-oss4h1 kernel: [1025097.221476] Lustre: atlas2-OST01a0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:10:13 atlas-oss4g6 kernel: [1025112.434300] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcce2c000 Dec 8 10:10:13 atlas-oss4g8 kernel: [1025107.987196] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc43f0000 Dec 8 10:10:13 atlas-oss4g8 kernel: [1025108.012456] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc43f0000 Dec 8 10:10:14 atlas-oss4g8 kernel: [1025108.051617] LustreError: 119646:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ffdabfc00 x1453430541504746/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386515708 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:10:14 atlas-oss4g8 kernel: [1025108.162375] Lustre: atlas2-OST03df: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:10:14 atlas-oss4h4 kernel: [438643.921268] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fabdf8000 Dec 8 10:10:14 atlas-oss4h4 kernel: [438643.945892] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fabdf8000 Dec 8 10:10:14 atlas-oss4h4 kernel: [438643.984971] LustreError: 61445:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880feaaa7800 x1453430541504742/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386515708 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:10:14 atlas-oss4h4 kernel: [438644.095662] Lustre: atlas2-OST03e3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:11:29 atlas-oss3b4 kernel: [1025181.115168] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88072d8ce000 Dec 8 10:11:29 atlas-oss3b4 kernel: [1025181.241372] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 10:11:30 atlas-oss3b4 kernel: [1025181.310487] LustreError: 11717:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8807847c8000 x1453425687731165/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386516069 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:11:30 atlas-oss3b4 kernel: [1025181.420243] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 10:13:51 atlas-oss4h1 kernel: [1025315.240237] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881072d24000 Dec 8 10:13:51 atlas-oss4h1 kernel: [1025315.268137] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdd2d4000 Dec 8 10:13:51 atlas-oss4h1 kernel: [1025315.307282] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881072d24000 Dec 8 10:13:51 atlas-oss4h1 kernel: [1025315.338230] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdd2d4000 Dec 8 10:15:49 atlas-oss4h4 kernel: [438978.660990] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc7dae000 Dec 8 10:15:49 atlas-oss4h4 kernel: [438978.690860] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbbc5a000 Dec 8 10:15:50 atlas-oss4h4 kernel: [438978.730000] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc7dae000 Dec 8 10:15:50 atlas-oss4h4 kernel: [438978.760859] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbbc5a000 Dec 8 10:16:30 atlas-oss3b4 kernel: [1025481.301116] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880759eb6000 Dec 8 10:18:48 atlas-oss4h1 kernel: [1025611.106956] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88103afd4000 Dec 8 10:18:48 atlas-oss4h1 kernel: [1025611.135934] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd7e78000 Dec 8 10:18:48 atlas-oss4h1 kernel: [1025611.174768] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff88103afd4000 Dec 8 10:18:48 atlas-oss4h1 kernel: [1025611.205799] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd7e78000 Dec 8 10:18:48 atlas-oss4g6 kernel: [1025626.324254] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fccc04000 Dec 8 10:18:48 atlas-oss4g6 kernel: [1025626.354960] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcd388000 Dec 8 10:18:48 atlas-oss4g6 kernel: [1025626.394226] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fccc04000 Dec 8 10:18:48 atlas-oss4g6 kernel: [1025626.425202] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcd388000 Dec 8 10:18:48 atlas-oss4g6 kernel: [1025626.425246] LustreError: 118578:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ffe4d2000 x1453430541518944/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386516507 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:18:48 atlas-oss4g6 kernel: [1025626.425353] Lustre: atlas2-OST03dd: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:18:51 atlas-oss4h4 kernel: [439160.068327] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbbc5a000 Dec 8 10:18:51 atlas-oss4h4 kernel: [439160.098148] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbbc5a000 Dec 8 10:18:52 atlas-oss4g7 kernel: [1025635.002924] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc752c000 Dec 8 10:18:52 atlas-oss4g7 kernel: [1025635.034771] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc752c000 Dec 8 10:18:53 atlas-oss4g8 kernel: [1025626.293859] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc1ce0000 Dec 8 10:18:53 atlas-oss4g8 kernel: [1025626.319825] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc1ce0000 Dec 8 10:21:31 atlas-oss3b4 kernel: [1025781.487055] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880730ee0000 Dec 8 10:21:31 atlas-oss3b4 kernel: [1025781.614001] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 10:21:32 atlas-oss3b4 kernel: [1025782.667223] LustreError: 11572:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880762a02400 x1453425687751837/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386516671 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:21:32 atlas-oss3b4 kernel: [1025782.778427] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 10:23:18 atlas-oss4h1 kernel: [1025881.058342] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd9892000 Dec 8 10:23:18 atlas-oss4h1 kernel: [1025881.090056] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88104ef3a000 Dec 8 10:23:18 atlas-oss4h1 kernel: [1025881.129212] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd9892000 Dec 8 10:23:18 atlas-oss4h1 kernel: [1025881.160184] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff88104ef3a000 Dec 8 10:23:18 atlas-oss4h1 kernel: [1025881.160215] LustreError: 113618:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff88104d358000 x1453430541525029/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386516491 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:23:19 atlas-oss4h1 kernel: [1025881.160338] Lustre: atlas2-OST03e0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:25:12 atlas-oss4h4 kernel: [439540.159871] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fb92b6000 Dec 8 10:25:12 atlas-oss4h4 kernel: [439540.185445] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbfbd0000 Dec 8 10:25:12 atlas-oss4h4 kernel: [439540.224575] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fb92b6000 Dec 8 10:25:12 atlas-oss4h4 kernel: [439540.255461] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbfbd0000 Dec 8 10:25:12 atlas-oss4h4 kernel: [439540.255493] LustreError: 44014:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ffa105000 x1453430541525031/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386516608 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:25:12 atlas-oss4h4 kernel: [439540.255618] Lustre: atlas2-OST01a3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:26:32 atlas-oss3b4 kernel: [1026081.672993] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88072999c000 Dec 8 10:28:17 atlas-oss4h1 kernel: [1026179.084674] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcb83a000 Dec 8 10:28:17 atlas-oss4h1 kernel: [1026179.113036] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd0656000 Dec 8 10:28:17 atlas-oss4h1 kernel: [1026179.152426] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcb83a000 Dec 8 10:28:17 atlas-oss4h1 kernel: [1026179.183449] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd0656000 Dec 8 10:28:18 atlas-oss4h4 kernel: [439725.886422] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc0bea000 Dec 8 10:28:18 atlas-oss4h4 kernel: [439725.913169] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc0bea000 Dec 8 10:28:19 atlas-oss4g7 kernel: [1026200.821028] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd17d0000 Dec 8 10:28:19 atlas-oss4g7 kernel: [1026200.852642] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd17d0000 Dec 8 10:28:19 atlas-oss4g7 kernel: [1026200.883554] LustreError: 114167:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff8810130db400 x1453430541536169/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386516791 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:28:19 atlas-oss4g7 kernel: [1026201.002805] Lustre: atlas2-OST03de: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:28:19 atlas-oss4g6 kernel: [1026196.461582] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcb350000 Dec 8 10:28:19 atlas-oss4g6 kernel: [1026196.484922] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdf39a000 Dec 8 10:28:19 atlas-oss4g6 kernel: [1026196.524049] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcb350000 Dec 8 10:28:19 atlas-oss4g6 kernel: [1026196.555170] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdf39a000 Dec 8 10:28:20 atlas-oss4g8 kernel: [1026192.111969] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcb60a000 Dec 8 10:28:20 atlas-oss4g8 kernel: [1026192.143727] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcb60a000 Dec 8 10:28:20 atlas-oss4g8 kernel: [1026192.174694] LustreError: 67567:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff881002a21400 x1453430541536170/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386516791 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:28:20 atlas-oss4g8 kernel: [1026192.294038] Lustre: atlas2-OST03df: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:31:33 atlas-oss3b4 kernel: [1026381.858942] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88080b89c000 Dec 8 10:31:33 atlas-oss3b4 kernel: [1026381.985436] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 10:31:33 atlas-oss3b4 kernel: [1026382.053918] LustreError: 11590:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880765191c00 x1453425687772564/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386517273 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:31:33 atlas-oss3b4 kernel: [1026382.161092] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 10:32:36 atlas-oss4h5 kernel: [1026444.518737] LustreError: 137-5: atlas2-OST02c0_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 10:32:43 atlas-oss4h1 kernel: [1026444.716843] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8810083a4000 Dec 8 10:32:43 atlas-oss4h1 kernel: [1026444.746501] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd7eb2000 Dec 8 10:32:43 atlas-oss4h1 kernel: [1026444.785866] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff8810083a4000 Dec 8 10:32:43 atlas-oss4h1 kernel: [1026444.816693] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd7eb2000 Dec 8 10:33:47 atlas-oss4h4 kernel: [440054.147306] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc8f54000 Dec 8 10:33:47 atlas-oss4h4 kernel: [440054.172539] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcd704000 Dec 8 10:33:47 atlas-oss4h4 kernel: [440054.211855] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc8f54000 Dec 8 10:33:47 atlas-oss4h4 kernel: [440054.242672] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcd704000 Dec 8 10:36:33 atlas-oss3b4 kernel: [1026682.044871] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880728e40000 Dec 8 10:37:42 atlas-oss4g6 kernel: [1026757.960463] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcd0fe000 Dec 8 10:37:42 atlas-oss4g6 kernel: [1026757.992230] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88105127a000 Dec 8 10:37:42 atlas-oss4g6 kernel: [1026758.033615] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcd0fe000 Dec 8 10:37:42 atlas-oss4g6 kernel: [1026758.062826] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff88105127a000 Dec 8 10:37:42 atlas-oss4g6 kernel: [1026758.062867] LustreError: 118586:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fe2f94800 x1453430541546304/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386517641 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:37:42 atlas-oss4g6 kernel: [1026758.063014] Lustre: atlas2-OST03dd: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:37:43 atlas-oss4g8 kernel: [1026753.610842] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc0602000 Dec 8 10:37:43 atlas-oss4g8 kernel: [1026753.638160] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc0602000 Dec 8 10:37:43 atlas-oss4h4 kernel: [440289.544917] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbdf26000 Dec 8 10:37:43 atlas-oss4h4 kernel: [440289.572947] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbdf26000 Dec 8 10:37:43 atlas-oss4h4 kernel: [440289.612289] LustreError: 61438:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880feb0ab000 x1453430541546308/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386517641 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:37:43 atlas-oss4h4 kernel: [440289.723024] Lustre: atlas2-OST03e3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:37:44 atlas-oss4g7 kernel: [1026764.479526] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc82e8000 Dec 8 10:37:44 atlas-oss4g7 kernel: [1026764.517161] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc82e8000 Dec 8 10:37:44 atlas-oss4h1 kernel: [1026744.902779] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcf0f6000 Dec 8 10:37:44 atlas-oss4h1 kernel: [1026744.934648] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcc5e6000 Dec 8 10:37:44 atlas-oss4h1 kernel: [1026744.965445] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcf0f6000 Dec 8 10:37:44 atlas-oss4h1 kernel: [1026745.004488] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcc5e6000 Dec 8 10:37:44 atlas-oss4h1 kernel: [1026745.004519] LustreError: 113587:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff881007352000 x1453430541546303/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386517358 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:37:44 atlas-oss4h1 kernel: [1026745.004618] Lustre: atlas2-OST01a0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:38:08 atlas-oss4h8 kernel: [1026778.352079] LustreError: 137-5: atlas2-OST0113_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 10:41:18 atlas-oss4h1 kernel: [1026958.704278] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcbabe000 Dec 8 10:41:18 atlas-oss4h1 kernel: [1026958.730979] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdd912000 Dec 8 10:41:18 atlas-oss4h1 kernel: [1026958.769938] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcbabe000 Dec 8 10:41:18 atlas-oss4h1 kernel: [1026958.800938] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdd912000 Dec 8 10:41:34 atlas-oss3b4 kernel: [1026982.230803] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88075c144000 Dec 8 10:41:34 atlas-oss3b4 kernel: [1026982.356040] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 10:41:35 atlas-oss3b4 kernel: [1026982.426639] LustreError: 11572:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88075d803000 x1453425687793460/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386517874 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:41:35 atlas-oss3b4 kernel: [1026982.531404] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 10:43:19 atlas-oss4h4 kernel: [440624.284631] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc29fc000 Dec 8 10:43:19 atlas-oss4h4 kernel: [440624.308194] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbf736000 Dec 8 10:43:19 atlas-oss4h4 kernel: [440624.347336] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc29fc000 Dec 8 10:43:19 atlas-oss4h4 kernel: [440624.378055] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbf736000 Dec 8 10:46:17 atlas-oss4h1 kernel: [1027256.730604] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd5a72000 Dec 8 10:46:17 atlas-oss4h1 kernel: [1027256.764547] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880ffd770000 Dec 8 10:46:17 atlas-oss4h1 kernel: [1027256.794026] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd5a72000 Dec 8 10:46:17 atlas-oss4h1 kernel: [1027256.833181] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880ffd770000 Dec 8 10:46:18 atlas-oss4g8 kernel: [1027267.598284] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fce6e6000 Dec 8 10:46:18 atlas-oss4g8 kernel: [1027267.625043] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fce6e6000 Dec 8 10:46:18 atlas-oss4g8 kernel: [1027267.664271] LustreError: 67539:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff3892000 x1453430541560505/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386518157 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:46:18 atlas-oss4g8 kernel: [1027267.775203] Lustre: atlas2-OST03df: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:46:22 atlas-oss4g6 kernel: [1027276.267121] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd03d4000 Dec 8 10:46:22 atlas-oss4g6 kernel: [1027276.294774] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcfea8000 Dec 8 10:46:22 atlas-oss4g6 kernel: [1027276.334138] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd03d4000 Dec 8 10:46:22 atlas-oss4g6 kernel: [1027276.364805] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcfea8000 Dec 8 10:46:23 atlas-oss4h4 kernel: [440807.851577] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc2960000 Dec 8 10:46:23 atlas-oss4h4 kernel: [440807.880976] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc2960000 Dec 8 10:46:23 atlas-oss4g7 kernel: [1027282.786177] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc7e22000 Dec 8 10:46:23 atlas-oss4g7 kernel: [1027282.817819] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc7e22000 Dec 8 10:46:23 atlas-oss4g7 kernel: [1027282.848530] LustreError: 115249:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff1489800 x1453430541560507/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386518157 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:46:23 atlas-oss4g7 kernel: [1027282.967484] Lustre: atlas2-OST03de: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:46:35 atlas-oss3b4 kernel: [1027282.416758] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880763e60000 Dec 8 10:50:38 atlas-oss4h5 kernel: [1027524.105347] LustreError: 137-5: atlas2-OST02c0_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 10:50:38 atlas-oss4g4 kernel: [1027528.991327] LustreError: 137-5: atlas2-OST007f_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 10:50:38 atlas-oss4h5 kernel: [1027524.130485] LustreError: 137-5: atlas2-OST0350_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 10:50:50 atlas-oss4h1 kernel: [1027528.841599] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd776a000 Dec 8 10:50:50 atlas-oss4h1 kernel: [1027528.873290] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fda1d6000 Dec 8 10:50:50 atlas-oss4h1 kernel: [1027528.912429] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd776a000 Dec 8 10:50:50 atlas-oss4h1 kernel: [1027528.943515] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fda1d6000 Dec 8 10:50:50 atlas-oss4h1 kernel: [1027528.943544] LustreError: 113545:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ffcc0d400 x1453430541566581/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386518141 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:50:50 atlas-oss4h1 kernel: [1027528.943651] Lustre: atlas2-OST03e0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:51:36 atlas-oss3b4 kernel: [1027582.602688] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88072b638000 Dec 8 10:51:36 atlas-oss3b4 kernel: [1027582.727486] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 10:51:36 atlas-oss3b4 kernel: [1027582.798345] LustreError: 11604:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880767260400 x1453425687814261/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386518476 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:51:36 atlas-oss3b4 kernel: [1027582.901395] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 10:52:44 atlas-oss4h4 kernel: [441187.943125] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbb688000 Dec 8 10:52:44 atlas-oss4h4 kernel: [441187.967932] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8810712d0000 Dec 8 10:52:44 atlas-oss4h4 kernel: [441188.007063] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbb688000 Dec 8 10:52:44 atlas-oss4h4 kernel: [441188.037970] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff8810712d0000 Dec 8 10:52:44 atlas-oss4h4 kernel: [441188.038001] LustreError: 61461:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fe73bc000 x1453430541710152/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386518541 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:52:44 atlas-oss4h4 kernel: [441188.038118] Lustre: atlas2-OST01a3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:52:57 atlas-oss1g8 kernel: [1027661.029160] LustreError: 46035:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff881017780e00 Dec 8 10:52:57 atlas-oss1g8 kernel: [1027661.053596] LustreError: 46035:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff881017780e00 Dec 8 10:52:57 atlas-oss1g8 kernel: [1027661.092974] LustreError: 115260:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(60243) req@ffff880fef3a1000 x1453430541749006/t0(0) o4->a2372b81-069b-e0b2-d123-6d252a16252e@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386518274 ref 1 fl Interpret:/0/0 rc 0/0 Dec 8 10:52:57 atlas-oss1g8 kernel: [1027661.096187] Lustre: atlas1-OST01e7: Client a2372b81-069b-e0b2-d123-6d252a16252e (at 10.36.202.142@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 10:52:57 atlas-oss1g8 kernel: [1027661.292306] Lustre: atlas1-OST01e7: Bulk IO write error with a2372b81-069b-e0b2-d123-6d252a16252e (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:55:45 atlas-oss4g8 kernel: [1027833.416382] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fe94e0000 Dec 8 10:55:45 atlas-oss4g8 kernel: [1027833.450051] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fe94e0000 Dec 8 10:55:48 atlas-oss4g7 kernel: [1027846.444676] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fca61e000 Dec 8 10:55:48 atlas-oss4g7 kernel: [1027846.471632] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fca61e000 Dec 8 10:55:48 atlas-oss4h1 kernel: [1027826.867934] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fb49be000 Dec 8 10:55:48 atlas-oss4h1 kernel: [1027826.896608] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd3220000 Dec 8 10:55:48 atlas-oss4h1 kernel: [1027826.935846] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fb49be000 Dec 8 10:55:48 atlas-oss4h1 kernel: [1027826.966709] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd3220000 Dec 8 10:55:49 atlas-oss4g6 kernel: [1027842.085229] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcbf6a000 Dec 8 10:55:49 atlas-oss4g6 kernel: [1027842.113730] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880feba52000 Dec 8 10:55:49 atlas-oss4g6 kernel: [1027842.153214] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcbf6a000 Dec 8 10:55:49 atlas-oss4g6 kernel: [1027842.184129] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880feba52000 Dec 8 10:55:49 atlas-oss4g6 kernel: [1027842.184160] LustreError: 61491:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff7fecc00 x1453430541769669/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386518724 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 10:55:49 atlas-oss4g6 kernel: [1027842.184290] Lustre: atlas2-OST03dd: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 10:55:50 atlas-oss4h4 kernel: [441373.669683] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbbc3a000 Dec 8 10:55:50 atlas-oss4h4 kernel: [441373.695827] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbbc3a000 Dec 8 10:56:37 atlas-oss3b4 kernel: [1027882.788647] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880761bae000 Dec 8 10:57:59 atlas-oss1b6 kernel: [1027968.019662] LustreError: 18905:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff88100f56a800 Dec 8 10:57:59 atlas-oss1b6 kernel: [1027968.045125] LustreError: 18905:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff88100f56a800 Dec 8 10:58:00 atlas-oss1b6 kernel: [1027968.084387] LustreError: 115317:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(471239) req@ffff880f851d9800 x1453430541775639/t0(0) o4->a2372b81-069b-e0b2-d123-6d252a16252e@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386518576 ref 1 fl Interpret:/0/0 rc 0/0 Dec 8 10:58:00 atlas-oss1b6 kernel: [1027968.088239] Lustre: atlas1-OST01bd: Client a2372b81-069b-e0b2-d123-6d252a16252e (at 10.36.202.142@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 10:58:00 atlas-oss1b6 kernel: [1027968.264745] Lustre: atlas1-OST01bd: Bulk IO write error with a2372b81-069b-e0b2-d123-6d252a16252e (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:00:12 atlas-oss4h1 kernel: [1028090.340483] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd3338000 Dec 8 11:00:12 atlas-oss4h1 kernel: [1028090.364924] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fca234000 Dec 8 11:00:12 atlas-oss4h1 kernel: [1028090.403977] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd3338000 Dec 8 11:00:12 atlas-oss4h1 kernel: [1028090.435005] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fca234000 Dec 8 11:01:19 atlas-oss4h4 kernel: [441701.930563] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcd694000 Dec 8 11:01:19 atlas-oss4h4 kernel: [441701.955381] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbd760000 Dec 8 11:01:19 atlas-oss4h4 kernel: [441701.994738] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcd694000 Dec 8 11:01:19 atlas-oss4h4 kernel: [441702.025375] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbd760000 Dec 8 11:01:38 atlas-oss3b4 kernel: [1028182.974571] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88071ce6c000 Dec 8 11:01:38 atlas-oss3b4 kernel: [1028183.098954] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 11:01:38 atlas-oss3b4 kernel: [1028183.154080] LustreError: 11604:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88086b813800 x1453425688080193/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386519078 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:01:38 atlas-oss3b4 kernel: [1028183.262787] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 11:05:12 atlas-oss4g8 kernel: [1028399.234491] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbfe30000 Dec 8 11:05:12 atlas-oss4g8 kernel: [1028399.263849] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbfe30000 Dec 8 11:05:12 atlas-oss4g8 kernel: [1028399.303203] LustreError: 121271:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff881002454800 x1453430541819155/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386519008 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:05:12 atlas-oss4g8 kernel: [1028399.414083] Lustre: atlas2-OST03df: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:05:12 atlas-oss4h4 kernel: [441935.168561] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbd084000 Dec 8 11:05:12 atlas-oss4h4 kernel: [441935.200373] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbd084000 Dec 8 11:05:12 atlas-oss4h4 kernel: [441935.239357] LustreError: 44014:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fe4ad2400 x1453430541819154/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386519008 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:05:13 atlas-oss4h4 kernel: [441935.350200] Lustre: atlas2-OST03e3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:05:13 atlas-oss4g7 kernel: [1028410.103169] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc81a8000 Dec 8 11:05:13 atlas-oss4g7 kernel: [1028410.134966] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc81a8000 Dec 8 11:05:13 atlas-oss4g7 kernel: [1028410.174161] LustreError: 115249:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880feb9b1400 x1453430541819166/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386519291 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:05:13 atlas-oss4g7 kernel: [1028410.293564] Lustre: atlas2-OST03de: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:05:13 atlas-oss4h1 kernel: [1028390.526428] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fddcf8000 Dec 8 11:05:13 atlas-oss4h1 kernel: [1028390.553422] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcbbce000 Dec 8 11:05:13 atlas-oss4h1 kernel: [1028390.592429] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fddcf8000 Dec 8 11:05:13 atlas-oss4h1 kernel: [1028390.623506] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcbbce000 Dec 8 11:05:13 atlas-oss4h1 kernel: [1028390.623535] LustreError: 42235:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff631b400 x1453430541819498/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386519008 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:05:13 atlas-oss4g6 kernel: [1028405.743730] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcbbc0000 Dec 8 11:05:13 atlas-oss4g6 kernel: [1028405.767708] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8810293e0000 Dec 8 11:05:13 atlas-oss4g6 kernel: [1028405.807101] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcbbc0000 Dec 8 11:05:13 atlas-oss4g6 kernel: [1028405.837836] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff8810293e0000 Dec 8 11:05:14 atlas-oss4h1 kernel: [1028390.623651] Lustre: atlas2-OST01a0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:05:40 atlas-oss4h8 kernel: [1028425.924570] LustreError: 137-5: atlas2-OST02c3_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 11:06:38 atlas-oss3b4 kernel: [1028483.160517] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880730e20000 Dec 8 11:08:52 atlas-oss4h1 kernel: [1028608.647144] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880ff20aa000 Dec 8 11:08:52 atlas-oss4h1 kernel: [1028608.679087] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd3af6000 Dec 8 11:08:52 atlas-oss4h1 kernel: [1028608.710150] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880ff20aa000 Dec 8 11:08:52 atlas-oss4h1 kernel: [1028608.749343] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd3af6000 Dec 8 11:10:50 atlas-oss4h4 kernel: [442272.067869] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbc37e000 Dec 8 11:10:50 atlas-oss4h4 kernel: [442272.091197] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbd9dc000 Dec 8 11:10:50 atlas-oss4h4 kernel: [442272.130422] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbc37e000 Dec 8 11:10:50 atlas-oss4h4 kernel: [442272.161440] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbd9dc000 Dec 8 11:11:39 atlas-oss3b4 kernel: [1028783.346456] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8807327ae000 Dec 8 11:11:39 atlas-oss3b4 kernel: [1028783.472190] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 11:11:40 atlas-oss3b4 kernel: [1028783.541780] LustreError: 11563:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8807638fec00 x1453425688369946/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386519679 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:11:40 atlas-oss3b4 kernel: [1028783.651964] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 11:13:48 atlas-oss4g7 kernel: [1028924.090609] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fccec0000 Dec 8 11:13:48 atlas-oss4g7 kernel: [1028924.115163] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fccec0000 Dec 8 11:13:49 atlas-oss4g6 kernel: [1028919.731159] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc7da6000 Dec 8 11:13:49 atlas-oss4g6 kernel: [1028919.758673] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc7da6000 Dec 8 11:13:49 atlas-oss4g6 kernel: [1028919.797984] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc9faa000 Dec 8 11:13:49 atlas-oss4g6 kernel: [1028919.798013] LustreError: 118688:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880febf67400 x1453430541834371/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386519807 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:13:49 atlas-oss4g6 kernel: [1028919.798123] Lustre: atlas2-OST03dd: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:13:49 atlas-oss4g6 kernel: [1028920.008271] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc9faa000 Dec 8 11:13:51 atlas-oss4g8 kernel: [1028917.541149] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbff92000 Dec 8 11:13:51 atlas-oss4g8 kernel: [1028917.571403] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbff92000 Dec 8 11:13:52 atlas-oss4h4 kernel: [442453.475226] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbc5ca000 Dec 8 11:13:52 atlas-oss4h4 kernel: [442453.498478] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbc5ca000 Dec 8 11:13:53 atlas-oss4h1 kernel: [1028908.833095] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880ffffcc000 Dec 8 11:13:53 atlas-oss4h1 kernel: [1028908.858347] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fe1142000 Dec 8 11:13:53 atlas-oss4h1 kernel: [1028908.897512] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880ffffcc000 Dec 8 11:13:53 atlas-oss4h1 kernel: [1028908.928667] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fe1142000 Dec 8 11:14:14 atlas-mds3 kernel: [1028156.561680] Lustre: 14622:0:(mdt_xattr.c:363:mdt_reint_setxattr()) atlas2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200026308:0xdc1:0x0]] [valid 68719476736] Dec 8 11:16:40 atlas-oss3b4 kernel: [1029083.532399] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88071be9c000 Dec 8 11:18:19 atlas-oss4h1 kernel: [1029174.465245] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcb4e6000 Dec 8 11:18:19 atlas-oss4h1 kernel: [1029174.491812] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881037dae000 Dec 8 11:18:19 atlas-oss4h1 kernel: [1029174.530973] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcb4e6000 Dec 8 11:18:19 atlas-oss4h1 kernel: [1029174.561965] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881037dae000 Dec 8 11:18:19 atlas-oss4h1 kernel: [1029174.561993] LustreError: 42253:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff88100a857800 x1453430541840453/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386519791 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:18:19 atlas-oss4h1 kernel: [1029174.562111] Lustre: atlas2-OST03e0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:20:19 atlas-oss4h4 kernel: [442840.045610] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fba80e000 Dec 8 11:20:19 atlas-oss4h4 kernel: [442840.071405] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc64a0000 Dec 8 11:20:19 atlas-oss4h4 kernel: [442840.110373] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fba80e000 Dec 8 11:20:19 atlas-oss4h4 kernel: [442840.141226] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc64a0000 Dec 8 11:20:19 atlas-oss4h4 kernel: [442840.141267] LustreError: 41413:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fe4c9c400 x1453430541843492/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386519910 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:20:19 atlas-oss4h4 kernel: [442840.141389] Lustre: atlas2-OST01a3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:21:41 atlas-oss3b4 kernel: [1029383.718337] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8807299b6000 Dec 8 11:21:41 atlas-oss3b4 kernel: [1029383.844165] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 11:21:42 atlas-oss3b4 kernel: [1029384.897381] LustreError: 11563:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8807616f4000 x1453425688632845/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386520281 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:21:42 atlas-oss3b4 kernel: [1029385.000603] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 11:23:18 atlas-oss4g8 kernel: [1029483.359249] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc77f0000 Dec 8 11:23:18 atlas-oss4g8 kernel: [1029483.385806] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc77f0000 Dec 8 11:23:18 atlas-oss4g8 kernel: [1029483.425105] LustreError: 67539:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ffd697400 x1453430541846537/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386520091 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:23:18 atlas-oss4g8 kernel: [1029483.536025] Lustre: atlas2-OST03df: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:23:19 atlas-oss4h4 kernel: [443019.293326] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc64a0000 Dec 8 11:23:19 atlas-oss4h4 kernel: [443019.323163] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc64a0000 Dec 8 11:23:19 atlas-oss4g7 kernel: [1029494.227935] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fddf12000 Dec 8 11:23:19 atlas-oss4g7 kernel: [1029494.254465] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fddf12000 Dec 8 11:23:20 atlas-oss4g7 kernel: [1029494.293821] LustreError: 113999:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff88106f5c1000 x1453430541846538/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386520091 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:23:20 atlas-oss4g7 kernel: [1029494.404728] Lustre: atlas2-OST03de: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:23:20 atlas-oss4h1 kernel: [1029474.651191] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fe07d0000 Dec 8 11:23:20 atlas-oss4h1 kernel: [1029474.680277] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880ff6410000 Dec 8 11:23:20 atlas-oss4h1 kernel: [1029474.719646] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fe07d0000 Dec 8 11:23:20 atlas-oss4h1 kernel: [1029474.750448] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880ff6410000 Dec 8 11:23:20 atlas-oss4g6 kernel: [1029489.868484] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcd878000 Dec 8 11:23:20 atlas-oss4g6 kernel: [1029489.898434] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd154e000 Dec 8 11:23:20 atlas-oss4g6 kernel: [1029489.937459] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcd878000 Dec 8 11:23:20 atlas-oss4g6 kernel: [1029489.968504] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd154e000 Dec 8 11:26:42 atlas-oss3b4 kernel: [1029683.904288] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88075aca4000 Dec 8 11:27:44 atlas-oss4h1 kernel: [1029738.123740] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcb4e6000 Dec 8 11:27:44 atlas-oss4h1 kernel: [1029738.148339] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd0648000 Dec 8 11:27:44 atlas-oss4h1 kernel: [1029738.187637] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcb4e6000 Dec 8 11:27:44 atlas-oss4h1 kernel: [1029738.218582] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd0648000 Dec 8 11:29:44 atlas-oss4h4 kernel: [443403.704101] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc893c000 Dec 8 11:29:44 atlas-oss4h4 kernel: [443403.730925] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbc6f0000 Dec 8 11:29:44 atlas-oss4h4 kernel: [443403.761817] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc893c000 Dec 8 11:29:44 atlas-oss4h4 kernel: [443403.800796] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbc6f0000 Dec 8 11:31:43 atlas-oss3b4 kernel: [1029984.090218] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880761db6000 Dec 8 11:31:43 atlas-oss3b4 kernel: [1029984.215921] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 11:31:44 atlas-oss3b4 kernel: [1029985.266092] LustreError: 11563:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880824f40400 x1453425688931359/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386520883 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:31:44 atlas-oss3b4 kernel: [1029985.370735] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 11:32:42 atlas-oss4g7 kernel: [1030055.726814] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcde3c000 Dec 8 11:32:42 atlas-oss4g7 kernel: [1030055.752646] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcde3c000 Dec 8 11:32:42 atlas-oss4h1 kernel: [1030036.150076] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdd740000 Dec 8 11:32:42 atlas-oss4h1 kernel: [1030036.181646] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880ff374c000 Dec 8 11:32:42 atlas-oss4h1 kernel: [1030036.220739] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdd740000 Dec 8 11:32:43 atlas-oss4h1 kernel: [1030036.251625] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880ff374c000 Dec 8 11:32:43 atlas-oss4h1 kernel: [1030036.251658] LustreError: 42253:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff6f93000 x1453430541863545/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386520658 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:32:43 atlas-oss4g6 kernel: [1030051.367369] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880ff798a000 Dec 8 11:32:43 atlas-oss4g6 kernel: [1030051.397091] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcecc0000 Dec 8 11:32:43 atlas-oss4g6 kernel: [1030051.436459] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880ff798a000 Dec 8 11:32:43 atlas-oss4g6 kernel: [1030051.467097] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcecc0000 Dec 8 11:32:43 atlas-oss4g6 kernel: [1030051.467127] LustreError: 118688:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff88101c567800 x1453430541863540/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386520941 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:32:43 atlas-oss4h1 kernel: [1030036.251756] Lustre: atlas2-OST01a0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:32:43 atlas-oss4g6 kernel: [1030051.467277] Lustre: atlas2-OST03dd: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:32:43 atlas-oss4g8 kernel: [1030047.017746] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc6a9e000 Dec 8 11:32:43 atlas-oss4g8 kernel: [1030047.044705] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc6a9e000 Dec 8 11:32:44 atlas-oss4h4 kernel: [443582.951818] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc75ce000 Dec 8 11:32:44 atlas-oss4h4 kernel: [443582.983017] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc75ce000 Dec 8 11:32:44 atlas-oss4h4 kernel: [443583.022045] LustreError: 41413:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fe1f29800 x1453430541863544/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386520941 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:32:44 atlas-oss4h4 kernel: [443583.132932] Lustre: atlas2-OST03e3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:36:23 atlas-oss4h1 kernel: [1030256.430400] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880ff5370000 Dec 8 11:36:23 atlas-oss4h1 kernel: [1030256.453693] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdc396000 Dec 8 11:36:23 atlas-oss4h1 kernel: [1030256.493052] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880ff5370000 Dec 8 11:36:23 atlas-oss4h1 kernel: [1030256.524153] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdc396000 Dec 8 11:36:46 atlas-oss3b4 kernel: [1030286.435787] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880741d44000 Dec 8 11:37:18 atlas-oss4h5 kernel: [1030318.093524] LustreError: 137-5: atlas2-OST0080_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 11:37:18 atlas-oss4g2 kernel: [1030321.801764] LustreError: 137-5: atlas2-OST007d_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 11:37:18 atlas-oss4g3 kernel: [1030320.822383] LustreError: 137-5: atlas2-OST007e_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 11:37:18 atlas-oss4h8 kernel: [1030320.587473] LustreError: 137-5: atlas2-OST0083_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 11:37:18 atlas-oss4h8 kernel: [1030320.587692] LustreError: 137-5: atlas2-OST0113_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 11:37:18 atlas-oss4g4 kernel: [1030322.990906] LustreError: 137-5: atlas2-OST007f_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 11:37:18 atlas-oss4g4 kernel: [1030322.991294] LustreError: 137-5: atlas2-OST019f_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 11:37:19 atlas-oss4g2 kernel: [1030322.800708] LustreError: 137-5: atlas2-OST02bd_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 11:37:19 atlas-oss4g4 kernel: [1030323.989928] LustreError: 137-5: atlas2-OST02bf_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 11:37:19 atlas-oss4g3 kernel: [1030321.821382] LustreError: 137-5: atlas2-OST02be_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 11:37:19 atlas-oss4h5 kernel: [1030319.092514] LustreError: 137-5: atlas2-OST02c0_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 11:37:19 atlas-oss4h8 kernel: [1030321.586484] LustreError: 137-5: atlas2-OST02c3_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 11:39:13 atlas-oss4h4 kernel: [443971.681815] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fa4db2000 Dec 8 11:39:13 atlas-oss4h4 kernel: [443971.711479] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc58ae000 Dec 8 11:39:13 atlas-oss4h4 kernel: [443971.750615] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fa4db2000 Dec 8 11:39:13 atlas-oss4h4 kernel: [443971.781448] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc58ae000 Dec 8 11:41:17 atlas-oss4g7 kernel: [1030569.714252] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fca300000 Dec 8 11:41:17 atlas-oss4g7 kernel: [1030569.743152] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fca300000 Dec 8 11:41:17 atlas-oss4g7 kernel: [1030569.782348] LustreError: 114122:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fedcc0400 x1453430541875681/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386521457 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:41:17 atlas-oss4g7 kernel: [1030569.893233] Lustre: atlas2-OST03de: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:41:22 atlas-oss4h1 kernel: [1030554.456735] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881037cc8000 Dec 8 11:41:22 atlas-oss4h1 kernel: [1030554.486475] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd3ed4000 Dec 8 11:41:22 atlas-oss4h1 kernel: [1030554.525613] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881037cc8000 Dec 8 11:41:22 atlas-oss4h1 kernel: [1030554.556649] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd3ed4000 Dec 8 11:41:22 atlas-oss4g6 kernel: [1030569.674028] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcb596000 Dec 8 11:41:22 atlas-oss4g6 kernel: [1030569.698514] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881051784000 Dec 8 11:41:22 atlas-oss4g6 kernel: [1030569.737723] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcb596000 Dec 8 11:41:22 atlas-oss4g6 kernel: [1030569.768651] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881051784000 Dec 8 11:41:23 atlas-oss4g8 kernel: [1030565.324409] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd0690000 Dec 8 11:41:23 atlas-oss4g8 kernel: [1030565.352428] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd0690000 Dec 8 11:41:23 atlas-oss4g8 kernel: [1030565.391663] LustreError: 67562:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fe75a0c00 x1453430541875683/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386521457 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:41:23 atlas-oss4g8 kernel: [1030565.502582] Lustre: atlas2-OST03df: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:41:23 atlas-oss4h4 kernel: [444101.258484] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc58ae000 Dec 8 11:41:23 atlas-oss4h4 kernel: [444101.290393] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc58ae000 Dec 8 11:41:46 atlas-oss3b4 kernel: [1030586.621711] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8807308ce000 Dec 8 11:41:47 atlas-oss3b4 kernel: [1030586.748424] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 11:41:48 atlas-oss3b4 kernel: [1030587.799057] LustreError: 11677:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8807626b3800 x1453425689203206/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386521487 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:41:48 atlas-oss3b4 kernel: [1030587.906477] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 11:45:50 atlas-oss4h1 kernel: [1030822.248509] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fda968000 Dec 8 11:45:50 atlas-oss4h1 kernel: [1030822.275445] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd0572000 Dec 8 11:45:50 atlas-oss4h1 kernel: [1030822.314418] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fda968000 Dec 8 11:45:50 atlas-oss4h1 kernel: [1030822.345480] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd0572000 Dec 8 11:45:50 atlas-oss4h1 kernel: [1030822.345509] LustreError: 42248:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff512cc00 x1453430541881785/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386521441 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:45:51 atlas-oss4h1 kernel: [1030822.345622] Lustre: atlas2-OST03e0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:46:47 atlas-oss3b4 kernel: [1030886.807649] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880732ac0000 Dec 8 11:48:38 atlas-oss4h4 kernel: [444535.340315] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc8418000 Dec 8 11:48:38 atlas-oss4h4 kernel: [444535.370962] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbd10e000 Dec 8 11:48:38 atlas-oss4h4 kernel: [444535.410128] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc8418000 Dec 8 11:48:38 atlas-oss4h4 kernel: [444535.441033] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbd10e000 Dec 8 11:48:38 atlas-oss4h4 kernel: [444535.441060] LustreError: 41043:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880fe1987800 x1453430541884824/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386521611 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:48:38 atlas-oss4h4 kernel: [444535.441217] Lustre: atlas2-OST01a3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:50:20 atlas-mds1 kernel: [398677.580287] Lustre: 13008:0:(mdt_xattr.c:363:mdt_reint_setxattr()) atlas1-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000208d2:0x5c2f:0x0]] [valid 68719476736] Dec 8 11:50:23 atlas-mds1 kernel: [398679.931106] Lustre: 13008:0:(mdt_xattr.c:363:mdt_reint_setxattr()) atlas1-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000208d2:0x5c90:0x0]] [valid 68719476736] Dec 8 11:50:49 atlas-oss4g7 kernel: [1031139.851580] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc752c000 Dec 8 11:50:49 atlas-oss4g7 kernel: [1031139.882336] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc752c000 Dec 8 11:50:49 atlas-oss4h1 kernel: [1031120.274834] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881037cc8000 Dec 8 11:50:49 atlas-oss4h1 kernel: [1031120.298564] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdd536000 Dec 8 11:50:49 atlas-oss4h1 kernel: [1031120.337756] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881037cc8000 Dec 8 11:50:49 atlas-oss4h1 kernel: [1031120.368627] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdd536000 Dec 8 11:50:49 atlas-oss4g6 kernel: [1031135.492128] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fca98c000 Dec 8 11:50:49 atlas-oss4g6 kernel: [1031135.517652] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd0b8a000 Dec 8 11:50:49 atlas-oss4g6 kernel: [1031135.556980] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fca98c000 Dec 8 11:50:49 atlas-oss4g6 kernel: [1031135.588006] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd0b8a000 Dec 8 11:50:49 atlas-oss4g6 kernel: [1031135.588039] LustreError: 118594:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff881072a44400 x1453430541887983/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386521741 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:50:49 atlas-oss4g6 kernel: [1031135.588157] Lustre: atlas2-OST03dd: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 11:50:50 atlas-oss4g8 kernel: [1031131.142512] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcf4c6000 Dec 8 11:50:50 atlas-oss4g8 kernel: [1031131.166894] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcf4c6000 Dec 8 11:50:50 atlas-oss4h4 kernel: [444667.076577] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc0822000 Dec 8 11:50:50 atlas-oss4h4 kernel: [444667.105793] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc0822000 Dec 8 11:51:48 atlas-oss3b4 kernel: [1031186.993604] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880730dae000 Dec 8 11:51:48 atlas-oss3b4 kernel: [1031187.117999] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 11:51:49 atlas-oss3b4 kernel: [1031188.170743] LustreError: 11572:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880764ce3400 x1453425689456747/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386522088 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 11:51:49 atlas-oss3b4 kernel: [1031188.267922] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 11:51:52 atlas-mds1 kernel: [398768.866864] Lustre: 13165:0:(mdt_xattr.c:363:mdt_reint_setxattr()) atlas1-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000208d2:0x5db1:0x0]] [valid 68719476736] Dec 8 11:53:07 atlas-mds1 kernel: [398843.380065] Lustre: 13042:0:(mdt_xattr.c:363:mdt_reint_setxattr()) atlas1-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000208d2:0x5f4f:0x0]] [valid 68719476736] Dec 8 11:53:48 atlas-mds1 kernel: [398884.632907] Lustre: 13042:0:(mdt_xattr.c:363:mdt_reint_setxattr()) atlas1-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000208d2:0x6145:0x0]] [valid 68719476736] Dec 8 11:55:13 atlas-oss4h1 kernel: [1031383.747393] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcf17a000 Dec 8 11:55:13 atlas-oss4h1 kernel: [1031383.787012] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff881037f08000 Dec 8 11:55:13 atlas-oss4h1 kernel: [1031383.816807] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcf17a000 Dec 8 11:55:13 atlas-oss4h1 kernel: [1031383.855994] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff881037f08000 Dec 8 11:55:37 atlas-mds1 kernel: [398993.484478] Lustre: 13003:0:(mdt_xattr.c:363:mdt_reint_setxattr()) atlas1-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000208d2:0x62dd:0x0]] [valid 68719476736] Dec 8 11:56:49 atlas-oss3b4 kernel: [1031487.179539] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88072c5e2000 Dec 8 11:58:05 atlas-oss4h4 kernel: [445101.158413] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fa39e4000 Dec 8 11:58:05 atlas-oss4h4 kernel: [445101.186199] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8810285d2000 Dec 8 11:58:05 atlas-oss4h4 kernel: [445101.225371] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fa39e4000 Dec 8 11:58:05 atlas-oss4h4 kernel: [445101.256359] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff8810285d2000 Dec 8 11:58:09 atlas-mds1 kernel: [399145.662425] Lustre: 13082:0:(mdt_xattr.c:363:mdt_reint_setxattr()) atlas1-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000208d2:0x6424:0x0]] [valid 68719476736] Dec 8 12:00:12 atlas-oss4g8 kernel: [1031692.641393] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8810308ba000 Dec 8 12:00:12 atlas-oss4g8 kernel: [1031692.670428] LustreError: 25559:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff8810308ba000 Dec 8 12:00:12 atlas-oss4g8 kernel: [1031692.709757] LustreError: 67544:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ffefa2800 x1453430541926172/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386522308 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 12:00:13 atlas-oss4g8 kernel: [1031692.820538] Lustre: atlas2-OST03df: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 12:00:13 atlas-oss4h4 kernel: [445228.575465] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8810712d0000 Dec 8 12:00:13 atlas-oss4h4 kernel: [445228.600309] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff8810712d0000 Dec 8 12:00:13 atlas-oss4h4 kernel: [445228.639403] LustreError: 61457:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff881004257800 x1453430541926173/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386522308 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 12:00:13 atlas-oss4h4 kernel: [445228.750155] Lustre: atlas2-OST03e3: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 12:00:14 atlas-oss4g7 kernel: [1031703.510071] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc9cda000 Dec 8 12:00:14 atlas-oss4g7 kernel: [1031703.535862] LustreError: 21296:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc9cda000 Dec 8 12:00:14 atlas-oss4g7 kernel: [1031703.575050] LustreError: 114085:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff880ff4646400 x1453430541926174/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386522308 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 12:00:14 atlas-oss4g7 kernel: [1031703.685787] Lustre: atlas2-OST03de: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 12:00:14 atlas-oss4h1 kernel: [1031683.933329] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd00f0000 Dec 8 12:00:14 atlas-oss4h1 kernel: [1031683.965182] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fdc974000 Dec 8 12:00:14 atlas-oss4h1 kernel: [1031684.004353] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd00f0000 Dec 8 12:00:14 atlas-oss4h1 kernel: [1031684.035061] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fdc974000 Dec 8 12:00:14 atlas-oss4h1 kernel: [1031684.035094] LustreError: 42248:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff881006d9a800 x1453430541926178/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386522308 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 12:00:14 atlas-oss4g6 kernel: [1031699.150625] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fceef0000 Dec 8 12:00:14 atlas-oss4g6 kernel: [1031699.181421] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc9ab8000 Dec 8 12:00:14 atlas-oss4g6 kernel: [1031699.220759] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fceef0000 Dec 8 12:00:14 atlas-oss4g6 kernel: [1031699.251783] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc9ab8000 Dec 8 12:00:14 atlas-oss4h1 kernel: [1031684.035223] Lustre: atlas2-OST01a0: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 12:01:50 atlas-oss3b4 kernel: [1031787.365479] LustreError: 25257:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88071b4a8000 Dec 8 12:01:50 atlas-oss3b4 kernel: [1031787.488315] Lustre: atlas2-OST009b: Client 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib) refused reconnection, still busy with 1 active RPCs Dec 8 12:01:51 atlas-oss3b4 kernel: [1031788.543421] LustreError: 11590:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88075c54cc00 x1453425689698153/t0(0) o3->3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2@10.36.202.138@o2ib:0/0 lens 448/432 e 0 to 0 dl 1386522690 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 12:01:51 atlas-oss3b4 kernel: [1031788.656915] Lustre: atlas2-OST009b: Bulk IO read error with 3b57a9ed-bec6-9b0d-7da8-04d696e1a7f2 (at 10.36.202.138@o2ib), client will retry: rc -110 Dec 8 12:02:29 atlas-mds1 kernel: [399405.032626] Lustre: 13048:0:(mdt_xattr.c:363:mdt_reint_setxattr()) atlas1-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000208d2:0x7a25:0x0]] [valid 68719476736] Dec 8 12:03:52 atlas-oss4h1 kernel: [1031902.054046] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd019a000 Dec 8 12:03:52 atlas-oss4h1 kernel: [1031902.081774] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcb70e000 Dec 8 12:03:52 atlas-oss4h1 kernel: [1031902.120970] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd019a000 Dec 8 12:03:52 atlas-oss4h1 kernel: [1031902.151950] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcb70e000 Dec 8 12:04:49 atlas-oss4g3 kernel: [1031968.310741] LustreError: 137-5: atlas2-OST007e_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 12:04:49 atlas-oss4h8 kernel: [1031968.064426] LustreError: 137-5: atlas2-OST0083_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 12:04:49 atlas-oss4g2 kernel: [1031969.291075] LustreError: 137-5: atlas2-OST007d_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 12:04:49 atlas-oss4h5 kernel: [1031965.579087] LustreError: 137-5: atlas2-OST0080_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 12:04:49 atlas-oss4g4 kernel: [1031970.483097] LustreError: 137-5: atlas2-OST007f_UUID: not available for connect from 10.36.202.142@o2ib (no target) Dec 8 12:06:51 atlas-oss3b4 kernel: [1032087.551421] LustreError: 25256:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88071afea000 Dec 8 12:07:11 atlas-mds3 kernel: [1031328.166681] Lustre: 14925:0:(mdt_xattr.c:363:mdt_reint_setxattr()) atlas2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000268c0:0xd4bf:0x0]] [valid 68719476736] Dec 8 12:07:32 atlas-oss4h4 kernel: [445666.976516] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fcde1c000 Dec 8 12:07:32 atlas-oss4h4 kernel: [445667.001095] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc232c000 Dec 8 12:07:32 atlas-oss4h4 kernel: [445667.040427] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fcde1c000 Dec 8 12:07:32 atlas-oss4h4 kernel: [445667.071432] LustreError: 16127:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc232c000 Dec 8 12:08:51 atlas-oss4h1 kernel: [1032200.080390] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fd3e78000 Dec 8 12:08:51 atlas-oss4h1 kernel: [1032200.105113] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880ff1fb8000 Dec 8 12:08:51 atlas-oss4h1 kernel: [1032200.143944] LustreError: 48032:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fd3e78000 Dec 8 12:08:51 atlas-oss4h1 kernel: [1032200.174786] LustreError: 48031:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880ff1fb8000 Dec 8 12:08:51 atlas-oss4g6 kernel: [1032215.297670] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8810518d6000 Dec 8 12:08:51 atlas-oss4g6 kernel: [1032215.327566] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fca46e000 Dec 8 12:08:51 atlas-oss4g6 kernel: [1032215.366704] LustreError: 28347:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff8810518d6000 Dec 8 12:08:51 atlas-oss4g6 kernel: [1032215.397871] LustreError: 28348:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fca46e000 Dec 8 12:08:51 atlas-oss4g6 kernel: [1032215.397905] LustreError: 118594:0:(ldlm_lib.c:2722:target_bulk_io()) @@@ network error on bulk GET 0(1048576) req@ffff881006230800 x1453430541943199/t0(0) o4->10c57078-ce10-72c6-b97d-e7c9a32a240c@10.36.202.142@o2ib:0/0 lens 448/448 e 0 to 0 dl 1386522824 ref 1 fl Interpret:/2/0 rc 0/0 Dec 8 12:08:52 atlas-oss4g6 kernel: [1032215.398020] Lustre: atlas2-OST03dd: Bulk IO write error with 10c57078-ce10-72c6-b97d-e7c9a32a240c (at 10.36.202.142@o2ib), client will retry: rc -110 Dec 8 12:08:52 atlas-oss4g8 kernel: [1032210.948054] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc5cea000 Dec 8 12:08:52 atlas-oss4g8 kernel: [1032210.977814] LustreError: 25558:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc5cea000 Dec 8 12:08:52 atlas-oss4h4 kernel: [445746.882127] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fbc3b8000 Dec 8 12:08:52 atlas-oss4h4 kernel: [445746.908083] LustreError: 16128:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fbc3b8000 Dec 8 12:08:53 atlas-oss4g7 kernel: [1032221.816734] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff880fc5bd8000 Dec 8 12:08:53 atlas-oss4g7 kernel: [1032221.847014] LustreError: 21297:0:(events.c:450:server_bulk_callback()) event type 3, status -5, desc ffff880fc5bd8000 Dec 8 12:10:45 atlas-mds1 kernel: [399900.266198] Lustre: 13020:0:(mdt_xattr.c:363:mdt_reint_setxattr()) atlas1-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000208d2:0x828e:0x0]] [valid 68719476736] Dec 8 12:10:57 atlas-oss1c1 kernel: [1032338.138464] sd 6:0:0:2: [sdc] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c1 kernel: [1032338.185775] sd 6:0:0:2: [sdc] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c6 kernel: [1032336.555638] sd 6:0:0:36: [sdi] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c4 kernel: [1032337.749976] sd 7:0:0:22: [sdad] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c2 kernel: [1032331.516293] sd 6:0:0:41: [sdn] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c3 kernel: [1032340.403366] sd 7:0:0:15: [sdb] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c3 kernel: [1032340.449645] sd 7:0:0:15: [sdb] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c6 kernel: [1032336.616160] sd 7:0:0:39: [sdan] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c8 kernel: [1032339.789756] sd 7:0:0:53: [sdan] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c8 kernel: [1032339.838736] sd 7:0:0:53: [sdan] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c8 kernel: [1032339.890033] sd 6:0:0:53: [sdl] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c2 kernel: [1032331.600400] sd 7:0:0:9: [sdae] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c7 kernel: [1032342.411283] sd 6:0:0:46: [sdan] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c7 kernel: [1032342.441784] sd 7:0:0:43: [sdi] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c7 kernel: [1032342.471887] sd 7:0:0:43: [sdi] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c5 kernel: [1032331.235318] sd 6:0:0:28: [sdh] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c5 kernel: [1032331.278898] sd 7:0:0:29: [sdak] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:57 atlas-oss1c5 kernel: [1032331.329617] sd 7:0:0:29: [sdak] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:58 atlas-oss1c4 kernel: [1032338.233701] sd 6:0:0:25: [sde] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:58 atlas-oss1c1 kernel: [1032338.765071] sd 7:0:0:3: [sdaf] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Dec 8 12:10:58 atlas-oss1c3 kernel: [1032341.022789] sd 6:0:0:18: [sdag] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.