Mar 16 02:14:24 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_19: cache:4236KB rss:6019632KB rss_huge:4003840KB mapped_file:4236KB swap:1268KB inactive_anon:781840KB active_anon:5242012KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:14:24 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_20: cache:4236KB rss:5980040KB rss_huge:3629056KB mapped_file:4236KB swap:46200KB inactive_anon:789440KB active_anon:5194788KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:14:24 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_21: cache:812KB rss:4818440KB rss_huge:3137536KB mapped_file:0KB swap:1209880KB inactive_anon:703704KB active_anon:4115740KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:14:24 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_22: cache:4236KB rss:6020412KB rss_huge:4280320KB mapped_file:4236KB swap:1144KB inactive_anon:858808KB active_anon:5165788KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:14:24 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_23: cache:4236KB rss:4692252KB rss_huge:3704832KB mapped_file:4236KB swap:1325412KB inactive_anon:687788KB active_anon:4008936KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:14:24 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_24: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:4180KB active_anon:4236KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:14:24 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_25: cache:4236KB rss:4826096KB rss_huge:3385344KB mapped_file:4236KB swap:1217756KB inactive_anon:682292KB active_anon:4148264KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:14:24 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_26: cache:4236KB rss:6075688KB rss_huge:4096000KB mapped_file:4236KB swap:612KB inactive_anon:852904KB active_anon:5226984KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:14:24 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_27: cache:872KB rss:5547484KB rss_huge:3459072KB mapped_file:0KB swap:521276KB inactive_anon:706472KB active_anon:4841972KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:14:24 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_28: cache:4236KB rss:6033732KB rss_huge:4337664KB mapped_file:4236KB swap:31544KB inactive_anon:728376KB active_anon:5309548KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:14:24 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_29: cache:4236KB rss:6010536KB rss_huge:3704832KB mapped_file:4236KB swap:53244KB inactive_anon:909240KB active_anon:5105508KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:14:24 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_30: cache:4236KB rss:6045720KB rss_huge:4030464KB mapped_file:4236KB swap:32796KB inactive_anon:847016KB active_anon:5202900KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:14:24 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_31: cache:4244KB rss:5485592KB rss_huge:4282368KB mapped_file:4244KB swap:89068KB inactive_anon:686472KB active_anon:4803340KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:14:24 john1 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Mar 16 02:14:24 john1 kernel: [ 5903] 0 5903 1079 86 8 0 0 sleep Mar 16 02:14:24 john1 kernel: [ 6160] 10056 6160 2414 294 10 93 0 slurm_script Mar 16 02:14:24 john1 kernel: [ 7231] 10056 7231 79435 494 34 699 0 srun Mar 16 02:14:24 john1 kernel: [ 7232] 10056 7232 11568 0 22 172 0 srun Mar 16 02:14:24 john1 kernel: [ 7249] 10056 7249 22610714 553824 1706 264996 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7250] 10056 7250 22605236 636404 1710 183387 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7251] 10056 7251 22605185 168137 1710 650677 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7252] 10056 7252 22605125 599957 1697 211495 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7253] 10056 7253 22605018 812784 1702 2329 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7254] 10056 7254 22603992 461669 1708 356720 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7255] 10056 7255 22604109 811104 1698 1485 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7256] 10056 7256 22604065 511346 1711 307915 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7257] 10056 7257 22604149 801209 1679 1597 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7258] 10056 7258 22604201 800806 1680 1731 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7259] 10056 7259 22604268 803166 1680 2526 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7260] 10056 7260 22604175 801934 1680 1521 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7261] 10056 7261 22699087 334457 1829 545288 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7262] 10056 7262 23307651 1497666 3053 9434 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7263] 10056 7263 23306626 1327065 3051 180287 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7264] 10056 7264 23305744 1503644 3049 1389 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7265] 10056 7265 23305824 1505082 3049 1830 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7266] 10056 7266 23307399 1417851 3054 88143 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7268] 10056 7268 23306398 1505715 3051 1459 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7269] 10056 7269 23307090 1413612 3053 94956 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7270] 10056 7270 23307003 1145011 3053 362205 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7271] 10056 7271 23306810 1506837 3052 508 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7272] 10056 7272 23307030 1115213 3053 391275 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7274] 10056 7274 23306167 1173341 3063 339692 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7275] 10056 7275 23306457 1520720 3079 361 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7276] 10056 7276 23307027 1297588 3073 219752 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7277] 10056 7277 23307205 1464038 3076 54276 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7278] 10056 7278 23307258 1437160 3078 80792 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7279] 10056 7279 23307528 1464807 3081 56796 0 measure_power_s Mar 16 02:14:24 john1 kernel: [ 7280] 10056 7280 23065060 1345135 2834 50557 0 measure_power_s Mar 16 02:14:24 john1 kernel: Memory cgroup out of memory: Kill process 7375 (measure_power_s) score 41 or sacrifice child Mar 16 02:14:24 john1 kernel: Killed process 7276 (measure_power_s) total-vm:93228108kB, anon-rss:5188324kB, file-rss:3164kB, shmem-rss:776kB Mar 16 02:14:56 john100 slurmd[1673]: task_p_slurmd_batch_request: 51014 Mar 16 02:14:56 john100 slurmd[1673]: task/affinity: job 51014 CPU input mask for node: 0xFFFFC3FFF Mar 16 02:14:56 john100 slurmd[1673]: task/affinity: job 51014 CPU final HW mask for node: 0xAAFFFFFFF Mar 16 02:14:57 john101 slurmd[1690]: _run_prolog: run job script took usec=86155 Mar 16 02:14:57 john101 slurmd[1690]: _run_prolog: prolog with lock for job 51014 ran for 1 seconds Mar 16 02:14:57 john100 slurmd[1673]: _run_prolog: run job script took usec=99977 Mar 16 02:14:57 john100 slurmd[1673]: _run_prolog: prolog with lock for job 51014 ran for 1 seconds Mar 16 02:14:58 john100 slurmd[1673]: Launching batch job 51014 for UID 10056 Mar 16 02:15:00 john1 kernel: measure_power_s invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0 Mar 16 02:15:00 john1 kernel: measure_power_s cpuset=step_0 mems_allowed=0-1 Mar 16 02:15:00 john1 kernel: CPU: 4 PID: 7253 Comm: measure_power_s Tainted: P OE ------------ T 3.10.0-693.21.1.el7.x86_64 #1 Mar 16 02:15:00 john1 kernel: Hardware name: Dell Inc. PowerEdge R740/06G98X, BIOS 1.3.7 02/08/2018 Mar 16 02:15:00 john1 kernel: Call Trace: Mar 16 02:15:00 john1 kernel: [] dump_stack+0x19/0x1b Mar 16 02:15:00 john1 kernel: [] dump_header+0x90/0x229 Mar 16 02:15:00 john1 kernel: [] ? default_wake_function+0x12/0x20 Mar 16 02:15:00 john1 kernel: [] ? find_lock_task_mm+0x56/0xc0 Mar 16 02:15:00 john1 kernel: [] ? try_get_mem_cgroup_from_mm+0x28/0x60 Mar 16 02:15:00 john1 kernel: [] oom_kill_process+0x254/0x3d0 Mar 16 02:15:00 john1 kernel: [] mem_cgroup_oom_synchronize+0x546/0x570 Mar 16 02:15:00 john1 kernel: [] ? mem_cgroup_charge_common+0xc0/0xc0 Mar 16 02:15:00 john1 kernel: [] pagefault_out_of_memory+0x14/0x90 Mar 16 02:15:00 john1 kernel: [] mm_fault_error+0x68/0x12b Mar 16 02:15:00 john1 kernel: [] __do_page_fault+0x391/0x450 Mar 16 02:15:00 john1 kernel: [] do_page_fault+0x35/0x90 Mar 16 02:15:00 john1 kernel: [] page_fault+0x28/0x30 Mar 16 02:15:00 john1 kernel: Task in /slurm/uid_10056/job_51013/step_0/task_30 killed as a result of limit of /slurm/uid_10056/job_51013 Mar 16 02:15:00 john1 kernel: memory: usage 134216112kB, limit 134217728kB, failcnt 3207176 Mar 16 02:15:00 john1 kernel: memory+swap: usage 147639500kB, limit 147639500kB, failcnt 79560 Mar 16 02:15:00 john1 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0 Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_extern: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_extern/task_0: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_batch: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_batch/task_0: cache:0KB rss:228KB rss_huge:0KB mapped_file:0KB swap:3528KB inactive_anon:228KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_0: cache:4252KB rss:2489232KB rss_huge:1347584KB mapped_file:4248KB swap:778048KB inactive_anon:692724KB active_anon:1800736KB inactive_file:4KB active_file:4KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_1: cache:4236KB rss:3804408KB rss_huge:2304000KB mapped_file:4236KB swap:558616KB inactive_anon:634828KB active_anon:3173816KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_2: cache:12KB rss:2157040KB rss_huge:272384KB mapped_file:12KB swap:1258984KB inactive_anon:1361788KB active_anon:795264KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john100 slurmd[1673]: launch task 51014.0 request from 10056.504@192.168.44.200 (port 50851) Mar 16 02:15:00 john101 slurmd[1690]: launch task 51014.0 request from 10056.504@192.168.44.200 (port 20626) Mar 16 02:15:00 john100 slurmd[1673]: lllp_distribution jobid [51014] implicit auto binding: cores,one_thread, dist 2 Mar 16 02:15:00 john100 slurmd[1673]: _task_layout_lllp_cyclic Mar 16 02:15:00 john101 slurmd[1690]: lllp_distribution jobid [51014] implicit auto binding: cores,one_thread, dist 2 Mar 16 02:15:00 john100 slurmd[1673]: _lllp_generate_cpu_bind jobid [51014]: mask_cpu,one_thread, 0x000000001,0x000000002,0x000000004,0x000000008,0x000000010,0x000000020,0x000000040,0x000000080,0x000000100,0x000000200,0x000000400,0x000000800,0x000001000,0x000002000,0x000004000,0x000008000,0x000010000,0x000020000,0x000040000,0x000080000,0x000100000,0x000200000,0x000400000,0x000800000,0x001000000,0x002000000,0x004000000,0x008000000,0x020000000,0x080000000,0x200000000,0x800000000 Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_3: cache:4236KB rss:3665536KB rss_huge:2125824KB mapped_file:4236KB swap:829428KB inactive_anon:614424KB active_anon:3055348KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john101 slurmd[1690]: _task_layout_lllp_cyclic Mar 16 02:15:00 john101 slurmd[1690]: _lllp_generate_cpu_bind jobid [51014]: mask_cpu,one_thread, 0x000000001,0x000000002,0x000000004,0x000000008,0x000000010,0x000000020,0x000000040,0x000000080,0x000000100,0x000000200,0x000000400,0x000000800,0x000001000,0x000002000,0x000004000,0x000008000,0x000010000,0x000020000,0x000040000,0x000080000,0x000100000,0x000200000,0x000400000,0x000800000,0x001000000,0x002000000,0x004000000,0x008000000,0x020000000,0x080000000,0x200000000,0x800000000 Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_4: cache:4276KB rss:1770056KB rss_huge:454656KB mapped_file:4276KB swap:1932952KB inactive_anon:474024KB active_anon:1300268KB inactive_file:0KB active_file:40KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_5: cache:596KB rss:2410424KB rss_huge:1427456KB mapped_file:0KB swap:1189260KB inactive_anon:606616KB active_anon:1804404KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_6: cache:4228KB rss:3806352KB rss_huge:2744320KB mapped_file:4228KB swap:811944KB inactive_anon:651972KB active_anon:3158608KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_7: cache:4212KB rss:2442604KB rss_huge:1482752KB mapped_file:4212KB swap:1136580KB inactive_anon:560552KB active_anon:1886264KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_8: cache:4248KB rss:3201760KB rss_huge:2050048KB mapped_file:4248KB swap:1544KB inactive_anon:675128KB active_anon:2530868KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_9: cache:4236KB rss:4519276KB rss_huge:3004416KB mapped_file:4236KB swap:1988KB inactive_anon:762428KB active_anon:3761084KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_10: cache:4236KB rss:4514224KB rss_huge:2809856KB mapped_file:4236KB swap:75004KB inactive_anon:795268KB active_anon:3723192KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_11: cache:4236KB rss:4464492KB rss_huge:3121152KB mapped_file:4236KB swap:1280KB inactive_anon:753148KB active_anon:3715580KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_12: cache:4236KB rss:2474676KB rss_huge:999424KB mapped_file:4236KB swap:1107908KB inactive_anon:1156748KB active_anon:1322164KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_13: cache:4236KB rss:5988148KB rss_huge:3424256KB mapped_file:4236KB swap:32296KB inactive_anon:749496KB active_anon:5242864KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_14: cache:4236KB rss:5677096KB rss_huge:3305472KB mapped_file:4236KB swap:344424KB inactive_anon:806836KB active_anon:4874476KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_15: cache:4236KB rss:6011240KB rss_huge:4012032KB mapped_file:4236KB swap:1056KB inactive_anon:806632KB active_anon:5208836KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_16: cache:4236KB rss:6019280KB rss_huge:3702784KB mapped_file:4236KB swap:600KB inactive_anon:901872KB active_anon:5121616KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:00 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_17: cache:4236KB rss:5989036KB rss_huge:3530752KB mapped_file:4236KB swap:27044KB inactive_anon:749164KB active_anon:5244064KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_18: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:2700KB active_anon:1536KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_19: cache:4236KB rss:6019632KB rss_huge:4003840KB mapped_file:4236KB swap:1268KB inactive_anon:781840KB active_anon:5242012KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_20: cache:4236KB rss:5980040KB rss_huge:3629056KB mapped_file:4236KB swap:46200KB inactive_anon:790464KB active_anon:5193764KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_21: cache:812KB rss:5149308KB rss_huge:3137536KB mapped_file:0KB swap:879180KB inactive_anon:1034380KB active_anon:4115740KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_22: cache:4236KB rss:6020412KB rss_huge:4280320KB mapped_file:4236KB swap:1144KB inactive_anon:858808KB active_anon:5165788KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_23: cache:4236KB rss:4969360KB rss_huge:3704832KB mapped_file:4236KB swap:1048492KB inactive_anon:964660KB active_anon:4008936KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_24: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:2180KB active_anon:2056KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_25: cache:4236KB rss:4900040KB rss_huge:3385344KB mapped_file:4236KB swap:1143992KB inactive_anon:756012KB active_anon:4148264KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_26: cache:4236KB rss:6075688KB rss_huge:4096000KB mapped_file:4236KB swap:612KB inactive_anon:852904KB active_anon:5226984KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_27: cache:872KB rss:0KB rss_huge:0KB mapped_file:0KB swap:3364KB inactive_anon:876KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_28: cache:4236KB rss:6033732KB rss_huge:4337664KB mapped_file:4236KB swap:31544KB inactive_anon:754744KB active_anon:5283180KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_29: cache:4236KB rss:6010536KB rss_huge:3704832KB mapped_file:4236KB swap:53244KB inactive_anon:909240KB active_anon:5105508KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_30: cache:4236KB rss:6045720KB rss_huge:4030464KB mapped_file:4236KB swap:32796KB inactive_anon:847016KB active_anon:5202900KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_31: cache:4244KB rss:5485592KB rss_huge:4282368KB mapped_file:4244KB swap:89068KB inactive_anon:686472KB active_anon:4803340KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Mar 16 02:15:01 john1 kernel: [ 5903] 0 5903 1079 86 8 0 0 sleep Mar 16 02:15:01 john1 kernel: [ 6160] 10056 6160 2414 294 10 93 0 slurm_script Mar 16 02:15:01 john1 kernel: [ 7231] 10056 7231 79435 494 34 699 0 srun Mar 16 02:15:01 john1 kernel: [ 7232] 10056 7232 11568 0 22 172 0 srun Mar 16 02:15:01 john1 kernel: [ 7249] 10056 7249 22610714 553450 1706 265367 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7250] 10056 7250 22605236 909335 2243 183387 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7251] 10056 7251 22605185 487584 1779 366325 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7252] 10056 7252 22605125 914221 2310 211495 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7253] 10056 7253 22605018 379575 1922 548166 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7254] 10056 7254 22603992 597779 1868 302178 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7255] 10056 7255 22604109 914582 2369 241960 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7256] 10056 7256 22604065 598077 1863 298656 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7257] 10056 7257 22604149 801239 1679 1597 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7258] 10056 7258 22604201 1130461 2324 1835 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7259] 10056 7259 22604268 954517 2351 194787 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7260] 10056 7260 22604175 1116863 2295 1521 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7261] 10056 7261 22699087 619452 1864 278176 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7262] 10056 7262 23307651 1497666 3053 9434 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7263] 10056 7263 23306626 1327065 3051 180287 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7264] 10056 7264 23305744 1503644 3049 1389 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7265] 10056 7265 23305824 1505082 3049 1830 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7266] 10056 7266 23307399 1417851 3054 88143 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7268] 10056 7268 23306398 1505715 3051 1459 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7269] 10056 7269 23307090 1413612 3053 94956 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7270] 10056 7270 23307003 1276325 3053 230891 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7271] 10056 7271 23306810 1506837 3052 508 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7272] 10056 7272 23307030 1243013 3053 263475 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7274] 10056 7274 23306167 1206845 3063 306188 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7275] 10056 7275 23306457 1520720 3079 361 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7277] 10056 7277 23307205 1464038 3076 54276 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7278] 10056 7278 23307258 1437160 3078 80792 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7279] 10056 7279 23307528 1464807 3081 56796 0 measure_power_s Mar 16 02:15:01 john1 kernel: [ 7280] 10056 7280 23065060 1345135 2834 50557 0 measure_power_s Mar 16 02:15:01 john1 kernel: Memory cgroup out of memory: Kill process 7356 (measure_power_s) score 41 or sacrifice child Mar 16 02:15:01 john1 kernel: Killed process 7279 (measure_power_s) total-vm:93230112kB, anon-rss:5851060kB, file-rss:3156kB, shmem-rss:5012kB Mar 16 02:15:01 john1 kernel: measure_power_s invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0 Mar 16 02:15:01 john1 kernel: measure_power_s cpuset=step_0 mems_allowed=0-1 Mar 16 02:15:01 john1 kernel: CPU: 10 PID: 7259 Comm: measure_power_s Tainted: P OE ------------ T 3.10.0-693.21.1.el7.x86_64 #1 Mar 16 02:15:01 john1 kernel: Hardware name: Dell Inc. PowerEdge R740/06G98X, BIOS 1.3.7 02/08/2018 Mar 16 02:15:01 john1 kernel: Call Trace: Mar 16 02:15:01 john1 kernel: [] dump_stack+0x19/0x1b Mar 16 02:15:01 john1 kernel: [] dump_header+0x90/0x229 Mar 16 02:15:01 john1 kernel: [] ? default_wake_function+0x12/0x20 Mar 16 02:15:01 john1 kernel: [] ? find_lock_task_mm+0x56/0xc0 Mar 16 02:15:01 john1 kernel: [] ? try_get_mem_cgroup_from_mm+0x28/0x60 Mar 16 02:15:01 john1 kernel: [] oom_kill_process+0x254/0x3d0 Mar 16 02:15:01 john1 kernel: [] mem_cgroup_oom_synchronize+0x546/0x570 Mar 16 02:15:01 john1 kernel: [] ? mem_cgroup_charge_common+0xc0/0xc0 Mar 16 02:15:01 john1 kernel: [] pagefault_out_of_memory+0x14/0x90 Mar 16 02:15:01 john1 kernel: [] mm_fault_error+0x68/0x12b Mar 16 02:15:01 john1 kernel: [] __do_page_fault+0x391/0x450 Mar 16 02:15:01 john1 kernel: [] do_page_fault+0x35/0x90 Mar 16 02:15:01 john1 kernel: [] page_fault+0x28/0x30 Mar 16 02:15:01 john1 kernel: Task in /slurm/uid_10056/job_51013/step_0/task_26 killed as a result of limit of /slurm/uid_10056/job_51013 Mar 16 02:15:01 john1 kernel: memory: usage 133711512kB, limit 134217728kB, failcnt 3208456 Mar 16 02:15:01 john1 kernel: memory+swap: usage 147214692kB, limit 147639500kB, failcnt 80151 Mar 16 02:15:01 john1 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0 Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_extern: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_extern/task_0: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_batch: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_batch/task_0: cache:0KB rss:228KB rss_huge:0KB mapped_file:0KB swap:3528KB inactive_anon:228KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:01 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_0: cache:4252KB rss:2487880KB rss_huge:1347584KB mapped_file:4248KB swap:779400KB inactive_anon:691372KB active_anon:1800736KB inactive_file:4KB active_file:4KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_1: cache:4236KB rss:4087476KB rss_huge:2578432KB mapped_file:4236KB swap:558616KB inactive_anon:644684KB active_anon:3447028KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_2: cache:12KB rss:2602868KB rss_huge:710656KB mapped_file:12KB swap:1258984KB inactive_anon:1366268KB active_anon:1248900KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_3: cache:4236KB rss:3784524KB rss_huge:2236416KB mapped_file:4236KB swap:829428KB inactive_anon:621332KB active_anon:3167428KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_4: cache:4276KB rss:2269196KB rss_huge:974848KB mapped_file:4276KB swap:1961384KB inactive_anon:476472KB active_anon:1811296KB inactive_file:0KB active_file:40KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_5: cache:596KB rss:2963356KB rss_huge:1972224KB mapped_file:0KB swap:1189260KB inactive_anon:606616KB active_anon:2371672KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_6: cache:4276KB rss:3797312KB rss_huge:2744320KB mapped_file:4264KB swap:820984KB inactive_anon:649760KB active_anon:3151780KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_7: cache:4212KB rss:3085032KB rss_huge:2117632KB mapped_file:4212KB swap:1136580KB inactive_anon:560552KB active_anon:2543028KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_8: cache:4248KB rss:3201760KB rss_huge:2050048KB mapped_file:4248KB swap:1544KB inactive_anon:675128KB active_anon:2530868KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_9: cache:4248KB rss:4576004KB rss_huge:3055616KB mapped_file:4248KB swap:1988KB inactive_anon:762428KB active_anon:3817812KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_10: cache:4236KB rss:4514224KB rss_huge:2809856KB mapped_file:4236KB swap:75004KB inactive_anon:795268KB active_anon:3723192KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_11: cache:4236KB rss:4580152KB rss_huge:3229696KB mapped_file:4236KB swap:1280KB inactive_anon:753148KB active_anon:3831240KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_12: cache:4236KB rss:2474676KB rss_huge:999424KB mapped_file:4236KB swap:1107908KB inactive_anon:1156748KB active_anon:1322164KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_13: cache:4236KB rss:5988148KB rss_huge:3424256KB mapped_file:4236KB swap:32296KB inactive_anon:749496KB active_anon:5242864KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_14: cache:4236KB rss:5677096KB rss_huge:3305472KB mapped_file:4236KB swap:344424KB inactive_anon:806836KB active_anon:4874476KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_15: cache:4236KB rss:6011240KB rss_huge:4012032KB mapped_file:4236KB swap:1056KB inactive_anon:806632KB active_anon:5208836KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_16: cache:4236KB rss:6019280KB rss_huge:3702784KB mapped_file:4236KB swap:600KB inactive_anon:901872KB active_anon:5121616KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_17: cache:4236KB rss:5989036KB rss_huge:3530752KB mapped_file:4236KB swap:27044KB inactive_anon:749164KB active_anon:5244064KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_18: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:2700KB active_anon:1536KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_19: cache:4236KB rss:6019632KB rss_huge:4003840KB mapped_file:4236KB swap:1268KB inactive_anon:781840KB active_anon:5242012KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_20: cache:4236KB rss:5980040KB rss_huge:3629056KB mapped_file:4236KB swap:46200KB inactive_anon:790464KB active_anon:5193764KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_21: cache:812KB rss:5149308KB rss_huge:3137536KB mapped_file:0KB swap:879180KB inactive_anon:1034380KB active_anon:4115740KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_22: cache:4236KB rss:6020412KB rss_huge:4280320KB mapped_file:4236KB swap:1144KB inactive_anon:858808KB active_anon:5165788KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_23: cache:4236KB rss:4969360KB rss_huge:3704832KB mapped_file:4236KB swap:1048492KB inactive_anon:964660KB active_anon:4008936KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_24: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:2180KB active_anon:2056KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_25: cache:4236KB rss:4900040KB rss_huge:3385344KB mapped_file:4236KB swap:1143992KB inactive_anon:756012KB active_anon:4148264KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_26: cache:4236KB rss:6075688KB rss_huge:4096000KB mapped_file:4236KB swap:612KB inactive_anon:852904KB active_anon:5226984KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_27: cache:872KB rss:0KB rss_huge:0KB mapped_file:0KB swap:3364KB inactive_anon:876KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_28: cache:4236KB rss:6033732KB rss_huge:4337664KB mapped_file:4236KB swap:31544KB inactive_anon:754744KB active_anon:5283180KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_29: cache:4236KB rss:6010536KB rss_huge:3704832KB mapped_file:4236KB swap:53244KB inactive_anon:909240KB active_anon:5105508KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_30: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:4176KB active_anon:4240KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_31: cache:4244KB rss:5485592KB rss_huge:4282368KB mapped_file:4244KB swap:89068KB inactive_anon:686472KB active_anon:4803340KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:02 john1 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Mar 16 02:15:02 john1 kernel: [ 5903] 0 5903 1079 86 8 0 0 sleep Mar 16 02:15:02 john1 kernel: [ 6160] 10056 6160 2414 294 10 93 0 slurm_script Mar 16 02:15:02 john1 kernel: [ 7231] 10056 7231 79435 494 34 699 0 srun Mar 16 02:15:02 john1 kernel: [ 7232] 10056 7232 11568 0 22 172 0 srun Mar 16 02:15:02 john1 kernel: [ 7249] 10056 7249 22610714 553125 1706 265692 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7250] 10056 7250 22605236 980142 2381 183387 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7251] 10056 7251 22605185 796541 2381 365962 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7252] 10056 7252 22605125 943998 2368 211495 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7253] 10056 7253 22605018 612786 2373 546214 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7254] 10056 7254 22603992 859949 2379 302159 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7255] 10056 7255 22604109 914614 2369 241960 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7256] 10056 7256 22604065 864215 2382 298656 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7257] 10056 7257 22604149 801239 1679 1597 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7258] 10056 7258 22604201 1144686 2351 1835 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7259] 10056 7259 22604268 954517 2351 194787 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7260] 10056 7260 22604175 1145820 2351 1521 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7261] 10056 7261 22699087 619452 1864 278176 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7262] 10056 7262 23307651 1497666 3053 9434 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7263] 10056 7263 23306626 1327065 3051 180287 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7264] 10056 7264 23305744 1503644 3049 1389 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7265] 10056 7265 23305824 1505082 3049 1830 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7266] 10056 7266 23307399 1417851 3054 88143 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7268] 10056 7268 23306398 1505715 3051 1459 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7269] 10056 7269 23307090 1413612 3053 94956 0 measure_power_s Mar 16 02:15:02 john1 kernel: [ 7270] 10056 7270 23307003 1276325 3053 230891 0 measure_power_s Mar 16 02:15:03 john1 kernel: [ 7271] 10056 7271 23306810 1506837 3052 508 0 measure_power_s Mar 16 02:15:03 john1 kernel: [ 7272] 10056 7272 23307030 1243013 3053 263475 0 measure_power_s Mar 16 02:15:03 john1 kernel: [ 7274] 10056 7274 23306167 1206845 3063 306188 0 measure_power_s Mar 16 02:15:03 john1 kernel: [ 7275] 10056 7275 23306457 1520720 3079 361 0 measure_power_s Mar 16 02:15:03 john1 kernel: [ 7277] 10056 7277 23307205 1464038 3076 54276 0 measure_power_s Mar 16 02:15:03 john1 kernel: [ 7278] 10056 7278 23307258 1437160 3078 80792 0 measure_power_s Mar 16 02:15:03 john1 kernel: [ 7280] 10056 7280 23065060 1345135 2834 50557 0 measure_power_s Mar 16 02:15:03 john1 kernel: Memory cgroup out of memory: Kill process 7365 (measure_power_s) score 41 or sacrifice child Mar 16 02:15:03 john1 kernel: Killed process 7275 (measure_power_s) total-vm:93225828kB, anon-rss:6074744kB, file-rss:3156kB, shmem-rss:4980kB Mar 16 02:15:45 john1 kernel: measure_power_s invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0 Mar 16 02:15:45 john1 kernel: measure_power_s cpuset=step_0 mems_allowed=0-1 Mar 16 02:15:45 john1 kernel: CPU: 4 PID: 7253 Comm: measure_power_s Tainted: P OE ------------ T 3.10.0-693.21.1.el7.x86_64 #1 Mar 16 02:15:45 john1 kernel: Hardware name: Dell Inc. PowerEdge R740/06G98X, BIOS 1.3.7 02/08/2018 Mar 16 02:15:45 john1 kernel: Call Trace: Mar 16 02:15:45 john1 kernel: [] dump_stack+0x19/0x1b Mar 16 02:15:45 john1 kernel: [] dump_header+0x90/0x229 Mar 16 02:15:45 john1 kernel: [] ? default_wake_function+0x12/0x20 Mar 16 02:15:45 john1 kernel: [] ? find_lock_task_mm+0x56/0xc0 Mar 16 02:15:45 john1 kernel: [] ? try_get_mem_cgroup_from_mm+0x28/0x60 Mar 16 02:15:45 john1 kernel: [] oom_kill_process+0x254/0x3d0 Mar 16 02:15:45 john1 kernel: [] mem_cgroup_oom_synchronize+0x546/0x570 Mar 16 02:15:45 john1 kernel: [] ? mem_cgroup_charge_common+0xc0/0xc0 Mar 16 02:15:45 john1 kernel: [] pagefault_out_of_memory+0x14/0x90 Mar 16 02:15:45 john1 kernel: [] mm_fault_error+0x68/0x12b Mar 16 02:15:45 john1 kernel: [] __do_page_fault+0x391/0x450 Mar 16 02:15:45 john1 kernel: [] do_page_fault+0x35/0x90 Mar 16 02:15:45 john1 kernel: [] page_fault+0x28/0x30 Mar 16 02:15:45 john1 kernel: Task in /slurm/uid_10056/job_51013/step_0/task_14 killed as a result of limit of /slurm/uid_10056/job_51013 Mar 16 02:15:45 john1 kernel: memory: usage 134198936kB, limit 134217728kB, failcnt 3258661 Mar 16 02:15:45 john1 kernel: memory+swap: usage 147639500kB, limit 147639500kB, failcnt 80844 Mar 16 02:15:46 john1 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0 Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_extern: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_extern/task_0: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_batch: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_batch/task_0: cache:0KB rss:156KB rss_huge:0KB mapped_file:0KB swap:3600KB inactive_anon:80KB active_anon:76KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_0: cache:4252KB rss:2487712KB rss_huge:1347584KB mapped_file:4248KB swap:779568KB inactive_anon:691204KB active_anon:1800736KB inactive_file:4KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_1: cache:4236KB rss:4087476KB rss_huge:2578432KB mapped_file:4236KB swap:558616KB inactive_anon:682060KB active_anon:3409652KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_2: cache:12KB rss:3392876KB rss_huge:1497088KB mapped_file:12KB swap:1257456KB inactive_anon:1378932KB active_anon:2013956KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_3: cache:4236KB rss:3784524KB rss_huge:2236416KB mapped_file:4236KB swap:829428KB inactive_anon:635404KB active_anon:3153356KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_4: cache:4276KB rss:3034988KB rss_huge:1232896KB mapped_file:4276KB swap:1592904KB inactive_anon:695748KB active_anon:2343476KB inactive_file:0KB active_file:40KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_5: cache:596KB rss:3459092KB rss_huge:2465792KB mapped_file:0KB swap:1189140KB inactive_anon:606736KB active_anon:2852952KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_6: cache:4232KB rss:3664612KB rss_huge:2744320KB mapped_file:4232KB swap:953684KB inactive_anon:643848KB active_anon:3024992KB inactive_file:0KB active_file:4KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_7: cache:4212KB rss:3506960KB rss_huge:2537472KB mapped_file:4212KB swap:1136528KB inactive_anon:626140KB active_anon:2885044KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_8: cache:4236KB rss:4577492KB rss_huge:3424256KB mapped_file:4236KB swap:1552KB inactive_anon:679984KB active_anon:3901744KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_9: cache:4236KB rss:4576004KB rss_huge:3055616KB mapped_file:4236KB swap:1988KB inactive_anon:762428KB active_anon:3817812KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_10: cache:4236KB rss:4514224KB rss_huge:2809856KB mapped_file:4236KB swap:75004KB inactive_anon:795268KB active_anon:3723192KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_11: cache:4236KB rss:4580152KB rss_huge:3229696KB mapped_file:4236KB swap:1280KB inactive_anon:753148KB active_anon:3831240KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_12: cache:4236KB rss:3850580KB rss_huge:2373632KB mapped_file:4236KB swap:1107804KB inactive_anon:1167464KB active_anon:2687356KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_13: cache:4236KB rss:6406352KB rss_huge:3727360KB mapped_file:4236KB swap:32296KB inactive_anon:801464KB active_anon:5609124KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_14: cache:4236KB rss:6905004KB rss_huge:4661248KB mapped_file:4236KB swap:491372KB inactive_anon:932572KB active_anon:5976668KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_15: cache:4236KB rss:6573340KB rss_huge:4419584KB mapped_file:4236KB swap:1056KB inactive_anon:937704KB active_anon:5639872KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_16: cache:4260KB rss:6019280KB rss_huge:3702784KB mapped_file:4256KB swap:600KB inactive_anon:901872KB active_anon:5121616KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_17: cache:4236KB rss:6384580KB rss_huge:3799040KB mapped_file:4236KB swap:27044KB inactive_anon:798700KB active_anon:5590116KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_18: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:2700KB active_anon:1536KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_19: cache:4236KB rss:6467968KB rss_huge:4306944KB mapped_file:4236KB swap:1268KB inactive_anon:847376KB active_anon:5624828KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_20: cache:4236KB rss:5980040KB rss_huge:3629056KB mapped_file:4236KB swap:46200KB inactive_anon:790464KB active_anon:5193764KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_21: cache:812KB rss:5248364KB rss_huge:3137536KB mapped_file:0KB swap:780124KB inactive_anon:1133436KB active_anon:4115740KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_22: cache:4236KB rss:6020412KB rss_huge:4280320KB mapped_file:4236KB swap:1144KB inactive_anon:858808KB active_anon:5165840KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_23: cache:4236KB rss:5067464KB rss_huge:3704832KB mapped_file:4236KB swap:950388KB inactive_anon:1062764KB active_anon:4008936KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_24: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:2180KB active_anon:2056KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_25: cache:4236KB rss:4976808KB rss_huge:3385344KB mapped_file:4236KB swap:1067224KB inactive_anon:832780KB active_anon:4148264KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_26: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:2180KB active_anon:2056KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_27: cache:872KB rss:0KB rss_huge:0KB mapped_file:0KB swap:3364KB inactive_anon:872KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_28: cache:4236KB rss:6033732KB rss_huge:4337664KB mapped_file:4236KB swap:31544KB inactive_anon:754744KB active_anon:5283180KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_29: cache:4236KB rss:6460144KB rss_huge:4005888KB mapped_file:4236KB swap:53244KB inactive_anon:909240KB active_anon:5555140KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_30: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:2176KB active_anon:2060KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:46 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_31: cache:4244KB rss:6015640KB rss_huge:4913152KB mapped_file:4244KB swap:467144KB inactive_anon:756848KB active_anon:5263028KB inactive_file:0KB active_file:8KB unevictable:0KB Mar 16 02:15:46 john1 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Mar 16 02:15:46 john1 kernel: [ 5903] 0 5903 1079 86 8 0 0 sleep Mar 16 02:15:46 john1 kernel: [ 6160] 10056 6160 2414 294 10 93 0 slurm_script Mar 16 02:15:46 john1 kernel: [ 7231] 10056 7231 79435 466 34 737 0 srun Mar 16 02:15:46 john1 kernel: [ 7232] 10056 7232 11568 0 22 172 0 srun Mar 16 02:15:46 john1 kernel: [ 7249] 10056 7249 22610714 553023 1706 265780 0 measure_power_s Mar 16 02:15:46 john1 kernel: [ 7250] 10056 7250 22605236 980139 2381 183387 0 measure_power_s Mar 16 02:15:46 john1 kernel: [ 7251] 10056 7251 22605185 796548 2381 365943 0 measure_power_s Mar 16 02:15:46 john1 kernel: [ 7252] 10056 7252 22605125 943995 2368 211495 0 measure_power_s Mar 16 02:15:46 john1 kernel: [ 7253] 10056 7253 22605018 720933 2373 438051 0 measure_power_s Mar 16 02:15:46 john1 kernel: [ 7254] 10056 7254 22603992 859950 2379 302146 0 measure_power_s Mar 16 02:15:46 john1 kernel: [ 7255] 10056 7255 22604109 909286 2369 247295 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7256] 10056 7256 22604065 864235 2382 298633 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7257] 10056 7257 22604149 1145193 2350 1598 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7258] 10056 7258 22604201 1144664 2351 1835 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7259] 10056 7259 22604268 954536 2351 194787 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7260] 10056 7260 22604175 1145835 2351 1521 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7261] 10056 7261 22699087 963481 2535 278136 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7262] 10056 7262 23307651 1602186 3257 9434 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7263] 10056 7263 23306626 1665966 3722 185142 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7264] 10056 7264 23305744 1644181 3323 1389 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7265] 10056 7265 23305824 1505127 3049 1830 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7266] 10056 7266 23307399 1516704 3247 88143 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7268] 10056 7268 23306398 1617825 3270 1459 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7269] 10056 7269 23307090 1413640 3053 94956 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7270] 10056 7270 23307003 1301123 3053 206096 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7271] 10056 7271 23306810 1506909 3052 508 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7272] 10056 7272 23307030 1267600 3053 238891 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7274] 10056 7274 23306167 1232088 3063 280948 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7277] 10056 7277 23307205 1464083 3076 54276 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7278] 10056 7278 23307258 1549551 3298 80792 0 measure_power_s Mar 16 02:15:47 john1 kernel: [ 7280] 10056 7280 23065060 1436181 3277 186563 0 measure_power_s Mar 16 02:15:47 john1 kernel: Memory cgroup out of memory: Kill process 7346 (measure_power_s) score 50 or sacrifice child Mar 16 02:15:47 john1 kernel: Killed process 7263 (measure_power_s) total-vm:93226504kB, anon-rss:6655772kB, file-rss:3204kB, shmem-rss:4888kB Mar 16 02:15:47 john1 kernel: measure_power_s invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0 Mar 16 02:15:47 john1 kernel: measure_power_s cpuset=step_0 mems_allowed=0-1 Mar 16 02:15:47 john1 kernel: CPU: 20 PID: 7269 Comm: measure_power_s Tainted: P OE ------------ T 3.10.0-693.21.1.el7.x86_64 #1 Mar 16 02:15:47 john1 kernel: Hardware name: Dell Inc. PowerEdge R740/06G98X, BIOS 1.3.7 02/08/2018 Mar 16 02:15:47 john1 kernel: Call Trace: Mar 16 02:15:47 john1 kernel: [] dump_stack+0x19/0x1b Mar 16 02:15:47 john1 kernel: [] dump_header+0x90/0x229 Mar 16 02:15:47 john1 kernel: [] ? default_wake_function+0x12/0x20 Mar 16 02:15:47 john1 kernel: [] ? find_lock_task_mm+0x56/0xc0 Mar 16 02:15:47 john1 kernel: [] ? try_get_mem_cgroup_from_mm+0x28/0x60 Mar 16 02:15:47 john1 kernel: [] oom_kill_process+0x254/0x3d0 Mar 16 02:15:47 john1 kernel: [] mem_cgroup_oom_synchronize+0x546/0x570 Mar 16 02:15:47 john1 kernel: [] ? mem_cgroup_charge_common+0xc0/0xc0 Mar 16 02:15:47 john1 kernel: [] pagefault_out_of_memory+0x14/0x90 Mar 16 02:15:47 john1 kernel: [] mm_fault_error+0x68/0x12b Mar 16 02:15:47 john1 kernel: [] __do_page_fault+0x391/0x450 Mar 16 02:15:47 john1 kernel: [] do_page_fault+0x35/0x90 Mar 16 02:15:47 john1 kernel: [] page_fault+0x28/0x30 Mar 16 02:15:47 john1 kernel: Task in /slurm/uid_10056/job_51013/step_0/task_15 killed as a result of limit of /slurm/uid_10056/job_51013 Mar 16 02:15:47 john1 kernel: memory: usage 132065868kB, limit 134217728kB, failcnt 3263527 Mar 16 02:15:47 john1 kernel: memory+swap: usage 145180956kB, limit 147639500kB, failcnt 81166 Mar 16 02:15:47 john1 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0 Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_extern: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_extern/task_0: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_batch: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_batch/task_0: cache:0KB rss:156KB rss_huge:0KB mapped_file:0KB swap:3600KB inactive_anon:80KB active_anon:76KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_0: cache:4252KB rss:2487712KB rss_huge:1347584KB mapped_file:4248KB swap:779568KB inactive_anon:691204KB active_anon:1800736KB inactive_file:4KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_1: cache:4244KB rss:4087476KB rss_huge:2578432KB mapped_file:4244KB swap:558616KB inactive_anon:682060KB active_anon:3409652KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_2: cache:24KB rss:3392876KB rss_huge:1497088KB mapped_file:24KB swap:1257456KB inactive_anon:1378932KB active_anon:2013956KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_3: cache:4236KB rss:3784524KB rss_huge:2236416KB mapped_file:4236KB swap:829428KB inactive_anon:635404KB active_anon:3153356KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_4: cache:4308KB rss:3017848KB rss_huge:1232896KB mapped_file:4308KB swap:1609976KB inactive_anon:668212KB active_anon:2353956KB inactive_file:0KB active_file:72KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_5: cache:600KB rss:3459092KB rss_huge:2465792KB mapped_file:4KB swap:1189140KB inactive_anon:606736KB active_anon:2852952KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_6: cache:4236KB rss:3652844KB rss_huge:2744320KB mapped_file:4236KB swap:965452KB inactive_anon:632080KB active_anon:3024992KB inactive_file:0KB active_file:4KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_7: cache:4212KB rss:3506960KB rss_huge:2537472KB mapped_file:4212KB swap:1136528KB inactive_anon:626140KB active_anon:2885044KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_8: cache:4240KB rss:4577492KB rss_huge:3424256KB mapped_file:4240KB swap:1552KB inactive_anon:679984KB active_anon:3901744KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_9: cache:4236KB rss:4576004KB rss_huge:3055616KB mapped_file:4236KB swap:1988KB inactive_anon:762428KB active_anon:3817812KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_10: cache:4240KB rss:4514224KB rss_huge:2809856KB mapped_file:4240KB swap:75004KB inactive_anon:795268KB active_anon:3723192KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_11: cache:4236KB rss:4580152KB rss_huge:3229696KB mapped_file:4236KB swap:1280KB inactive_anon:753148KB active_anon:3831240KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_12: cache:4236KB rss:3850580KB rss_huge:2373632KB mapped_file:4236KB swap:1107804KB inactive_anon:1167464KB active_anon:2687356KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_13: cache:4236KB rss:7082748KB rss_huge:4100096KB mapped_file:4236KB swap:32296KB inactive_anon:809912KB active_anon:6286236KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_14: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:4184KB active_anon:4232KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_15: cache:4236KB rss:7299108KB rss_huge:4792320KB mapped_file:4236KB swap:1056KB inactive_anon:937704KB active_anon:6374752KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_16: cache:4260KB rss:6019280KB rss_huge:3702784KB mapped_file:4256KB swap:600KB inactive_anon:901872KB active_anon:5121616KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:47 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_17: cache:4236KB rss:7161752KB rss_huge:4171776KB mapped_file:4236KB swap:27044KB inactive_anon:806636KB active_anon:6368596KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_18: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:2700KB active_anon:1536KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_19: cache:4236KB rss:7292592KB rss_huge:4679680KB mapped_file:4236KB swap:1268KB inactive_anon:847376KB active_anon:6458656KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_20: cache:4236KB rss:5980040KB rss_huge:3629056KB mapped_file:4236KB swap:46200KB inactive_anon:790464KB active_anon:5193812KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_21: cache:812KB rss:5248364KB rss_huge:3137536KB mapped_file:0KB swap:780124KB inactive_anon:1133436KB active_anon:4115740KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_22: cache:4236KB rss:7299104KB rss_huge:5545984KB mapped_file:4236KB swap:1144KB inactive_anon:858808KB active_anon:6458868KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_23: cache:4236KB rss:5257324KB rss_huge:3704832KB mapped_file:4236KB swap:950388KB inactive_anon:1062764KB active_anon:4205164KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_24: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:2180KB active_anon:2056KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_25: cache:4236KB rss:5911592KB rss_huge:3751936KB mapped_file:4236KB swap:1067224KB inactive_anon:832780KB active_anon:5084688KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_26: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:2180KB active_anon:2056KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_27: cache:872KB rss:0KB rss_huge:0KB mapped_file:0KB swap:3364KB inactive_anon:872KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_28: cache:4236KB rss:6033732KB rss_huge:4337664KB mapped_file:4236KB swap:31544KB inactive_anon:754744KB active_anon:5283180KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_29: cache:4236KB rss:7385836KB rss_huge:4376576KB mapped_file:4236KB swap:53244KB inactive_anon:909240KB active_anon:6480816KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_30: cache:4236KB rss:0KB rss_huge:0KB mapped_file:36KB swap:0KB inactive_anon:2176KB active_anon:2060KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 16 02:15:48 john1 kernel: Memory cgroup stats for /slurm/uid_10056/job_51013/step_0/task_31: cache:4244KB rss:5939496KB rss_huge:4913152KB mapped_file:4244KB swap:543548KB inactive_anon:748944KB active_anon:5194400KB inactive_file:0KB active_file:8KB unevictable:0KB Mar 16 02:15:48 john1 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Mar 16 02:15:48 john1 kernel: [ 5903] 0 5903 1079 86 8 0 0 sleep Mar 16 02:15:48 john1 kernel: [ 6160] 10056 6160 2414 294 10 93 0 slurm_script Mar 16 02:15:48 john1 kernel: [ 7231] 10056 7231 79435 466 34 737 0 srun Mar 16 02:15:48 john1 kernel: [ 7232] 10056 7232 11568 0 22 172 0 srun Mar 16 02:15:48 john1 kernel: [ 7249] 10056 7249 22610714 553023 1706 265780 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7250] 10056 7250 22605236 980150 2381 183387 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7251] 10056 7251 22605185 796557 2381 365943 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7252] 10056 7252 22605125 943995 2368 211495 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7253] 10056 7253 22605018 723371 2373 435625 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7254] 10056 7254 22603992 859950 2379 302146 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7255] 10056 7255 22604109 902924 2369 253668 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7256] 10056 7256 22604065 864241 2382 298633 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7257] 10056 7257 22604149 1145185 2350 1598 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7258] 10056 7258 22604201 1105918 2351 40581 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7259] 10056 7259 22604268 954548 2351 194787 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7260] 10056 7260 22604175 1145835 2351 1521 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7261] 10056 7261 22699087 963491 2535 278136 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7262] 10056 7262 23307651 1839324 3720 9434 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7264] 10056 7264 23305744 1847525 3720 1389 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7265] 10056 7265 23305824 1505127 3049 1830 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7266] 10056 7266 23307399 1753056 3708 88143 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7268] 10056 7268 23306398 1849488 3722 1459 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7269] 10056 7269 23307090 1413640 3053 94956 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7270] 10056 7270 23307003 1301123 3053 206096 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7271] 10056 7271 23306810 1841752 3706 508 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7272] 10056 7272 23307030 1319300 3154 238891 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7274] 10056 7274 23306167 1467150 3522 280948 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7277] 10056 7277 23307205 1464083 3076 54276 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7278] 10056 7278 23307258 1780962 3749 80792 0 measure_power_s Mar 16 02:15:48 john1 kernel: [ 7280] 10056 7280 23065060 1416311 3277 206433 0 measure_power_s Mar 16 02:15:48 john1 kernel: Memory cgroup out of memory: Kill process 7373 (measure_power_s) score 44 or sacrifice child Mar 16 02:15:48 john1 kernel: Killed process 7264 (measure_power_s) total-vm:93222976kB, anon-rss:7382044kB, file-rss:3168kB, shmem-rss:4888kB Mar 16 02:19:42 john100 slurmd[1673]: task_p_slurmd_batch_request: 51015 Mar 16 02:19:42 john100 slurmd[1673]: task/affinity: job 51015 CPU input mask for node: 0xFFFFC3FFF Mar 16 02:19:42 john100 slurmd[1673]: task/affinity: job 51015 CPU final HW mask for node: 0xAAFFFFFFF Mar 16 02:19:42 john101 slurmd[1690]: _run_prolog: run job script took usec=86265 Mar 16 02:19:42 john101 slurmd[1690]: _run_prolog: prolog with lock for job 51015 ran for 0 seconds Mar 16 02:19:42 john100 slurmd[1673]: _run_prolog: run job script took usec=90796 Mar 16 02:19:42 john100 slurmd[1673]: _run_prolog: prolog with lock for job 51015 ran for 0 seconds Mar 16 02:19:43 john100 slurmd[1673]: Launching batch job 51015 for UID 10056 Mar 16 02:19:46 john101 slurmd[1690]: launch task 51015.0 request from 10056.504@192.168.44.200 (port 33938) Mar 16 02:19:46 john100 slurmd[1673]: launch task 51015.0 request from 10056.504@192.168.44.200 (port 64163) Mar 16 02:19:46 john101 slurmd[1690]: lllp_distribution jobid [51015] implicit auto binding: cores,one_thread, dist 2 Mar 16 02:19:46 john100 slurmd[1673]: lllp_distribution jobid [51015] implicit auto binding: cores,one_thread, dist 2 Mar 16 02:19:46 john101 slurmd[1690]: _task_layout_lllp_cyclic Mar 16 02:19:46 john100 slurmd[1673]: _task_layout_lllp_cyclic Mar 16 02:19:46 john101 slurmd[1690]: _lllp_generate_cpu_bind jobid [51015]: mask_cpu,one_thread, 0x000000001,0x000000002,0x000000004,0x000000008,0x000000010,0x000000020,0x000000040,0x000000080,0x000000100,0x000000200,0x000000400,0x000000800,0x000001000,0x000002000,0x000004000,0x000008000,0x000010000,0x000020000,0x000040000,0x000080000,0x000100000,0x000200000,0x000400000,0x000800000,0x001000000,0x002000000,0x004000000,0x008000000,0x020000000,0x080000000,0x200000000,0x800000000 Mar 16 02:19:46 john100 slurmd[1673]: _lllp_generate_cpu_bind jobid [51015]: mask_cpu,one_thread, 0x000000001,0x000000002,0x000000004,0x000000008,0x000000010,0x000000020,0x000000040,0x000000080,0x000000100,0x000000200,0x000000400,0x000000800,0x000001000,0x000002000,0x000004000,0x000008000,0x000010000,0x000020000,0x000040000,0x000080000,0x000100000,0x000200000,0x000400000,0x000800000,0x001000000,0x002000000,0x004000000,0x008000000,0x020000000,0x080000000,0x200000000,0x800000000 Mar 16 02:22:59 john2 slurmd[1681]: task_p_slurmd_batch_request: 51016 Mar 16 02:22:59 john2 slurmd[1681]: task/affinity: job 51016 CPU input mask for node: 0xFFFFC3FFF Mar 16 02:22:59 john2 slurmd[1681]: task/affinity: job 51016 CPU final HW mask for node: 0xAAFFFFFFF Mar 16 02:23:00 john2 slurmd[1681]: _run_prolog: run job script took usec=305863 Mar 16 02:23:00 john2 slurmd[1681]: _run_prolog: prolog with lock for job 51016 ran for 1 seconds Mar 16 02:23:01 john2 slurmd[1681]: Launching batch job 51016 for UID 10056 Mar 16 02:23:03 john2 slurmd[1681]: launch task 51016.0 request from 10056.504@192.168.44.102 (port 12492) Mar 16 02:23:03 john2 slurmd[1681]: lllp_distribution jobid [51016] implicit auto binding: cores,one_thread, dist 2 Mar 16 02:23:03 john2 slurmd[1681]: _task_layout_lllp_cyclic Mar 16 02:23:03 john2 slurmd[1681]: _lllp_generate_cpu_bind jobid [51016]: mask_cpu,one_thread, 0x000000001,0x000000002,0x000000004,0x000000008,0x000000010,0x000000020,0x000000040,0x000000080,0x000000100,0x000000200,0x000000400,0x000000800,0x000001000,0x000002000,0x000004000,0x000008000,0x000010000,0x000020000,0x000040000,0x000080000,0x000100000,0x000200000,0x000400000,0x000800000,0x001000000,0x002000000,0x004000000,0x008000000,0x020000000,0x080000000,0x200000000,0x800000000 Mar 16 02:24:25 john4 slurmd[1677]: task_p_slurmd_batch_request: 51017 Mar 16 02:24:25 john4 slurmd[1677]: task/affinity: job 51017 CPU input mask for node: 0xFFC003F00 Mar 16 02:24:25 john4 slurmd[1677]: task/affinity: job 51017 CPU final HW mask for node: 0xAAFFF0000 Mar 16 02:24:26 john4 slurmd[1677]: _run_prolog: run job script took usec=83625 Mar 16 02:24:26 john4 slurmd[1677]: _run_prolog: prolog with lock for job 51017 ran for 1 seconds Mar 16 02:24:27 john4 slurmd[1677]: Launching batch job 51017 for UID 10056 Mar 16 02:24:29 john4 slurmd[1677]: launch task 51017.0 request from 10056.504@192.168.44.104 (port 22164) Mar 16 02:24:29 john4 slurmd[1677]: lllp_distribution jobid [51017] implicit auto binding: cores,one_thread, dist 2 Mar 16 02:24:29 john4 slurmd[1677]: _task_layout_lllp_cyclic Mar 16 02:24:29 john4 slurmd[1677]: _lllp_generate_cpu_bind jobid [51017]: mask_cpu,one_thread, 0x000010000,0x000020000,0x000040000,0x000080000,0x000100000,0x000200000,0x000400000,0x000800000,0x001000000,0x002000000,0x004000000,0x008000000,0x020000000,0x080000000,0x200000000,0x800000000 Mar 16 02:26:05 john2 slurmd[1681]: task_p_slurmd_batch_request: 51018 Mar 16 02:26:05 john2 slurmd[1681]: task/affinity: job 51018 CPU input mask for node: 0xFFFFC3FFF Mar 16 02:26:05 john2 slurmd[1681]: task/affinity: job 51018 CPU final HW mask for node: 0xAAFFFFFFF Mar 16 02:26:05 john2 slurmd[1681]: _run_prolog: run job script took usec=90112 Mar 16 02:26:05 john2 slurmd[1681]: _run_prolog: prolog with lock for job 51018 ran for 0 seconds Mar 16 02:26:06 john2 slurmd[1681]: Launching batch job 51018 for UID 10056 Mar 16 02:26:08 john2 slurmd[1681]: launch task 51018.0 request from 10056.504@192.168.44.102 (port 23244) Mar 16 02:26:08 john2 slurmd[1681]: lllp_distribution jobid [51018] implicit auto binding: cores,one_thread, dist 2 Mar 16 02:26:08 john2 slurmd[1681]: _task_layout_lllp_cyclic Mar 16 02:26:08 john2 slurmd[1681]: _lllp_generate_cpu_bind jobid [51018]: mask_cpu,one_thread, 0x000000001,0x000000002,0x000000004,0x000000008,0x000000010,0x000000020,0x000000040,0x000000080,0x000000100,0x000000200,0x000000400,0x000000800,0x000001000,0x000002000,0x000004000,0x000008000,0x000010000,0x000020000,0x000040000,0x000080000,0x000100000,0x000200000,0x000400000,0x000800000,0x001000000,0x002000000,0x004000000,0x008000000,0x020000000,0x080000000,0x200000000,0x800000000 Mar 16 02:57:01 transom1 dhcpd: Remove host declaration arkle5.oob.sut or remove 10.7.2.35 Mar 16 03:16:50 john24 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448284013248 length 1048576 too big: 1046748 left, 1046748 allowed Mar 16 03:16:50 arkle7 kernel: LNet: Using FMR for registration Mar 16 03:16:50 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c7ffaea00 Mar 16 03:16:50 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c7ffaea00 Mar 16 03:16:50 arkle7 kernel: LustreError: 34897:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881307765c50 x1594448284013248/t0(0) o4->6ec8b14c-8574-1b1c-979a-d89152f9126c@192.168.44.124@o2ib44:443/0 lens 608/448 e 0 to 0 dl 1521130653 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:50 arkle7 kernel: Lustre: dagg-OST000c: Bulk IO write error with 6ec8b14c-8574-1b1c-979a-d89152f9126c (at 192.168.44.124@o2ib44), client will retry: rc = -110 Mar 16 03:16:50 arkle7 kernel: LNet: Skipped 1 previous similar message Mar 16 03:16:50 arkle4 kernel: LustreError: 168-f: dagg-OST0006: BAD WRITE CHECKSUM: from 12345-192.168.44.132@o2ib44 inode [0x200010677:0x17f:0x0] object 0x0:8451333 extent [2671290628-2675482623]: client csum c582eef7, server csum 54b74bd9 Mar 16 03:16:50 arkle6 kernel: LustreError: 168-f: dagg-OST000a: BAD WRITE CHECKSUM: from 12345-192.168.44.109@o2ib44 inode [0x200010631:0x3b9:0x0] object 0x0:8426505 extent [2509346524-2513539071]: client csum 42f9c388, server csum 9fccde87 Mar 16 03:16:50 john7 kernel: LNetError: 914:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448319400640 length 1048576 too big: 1045464 left, 1045464 allowed Mar 16 03:16:50 john14 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448317079152 length 1048576 too big: 1047896 left, 1047896 allowed Mar 16 03:16:50 arkle8 kernel: LustreError: 168-f: dagg-OST000f: BAD WRITE CHECKSUM: from 12345-192.168.44.132@o2ib44 inode [0x68001c9cb:0x1d3:0x0] object 0x380000401:9525425 extent [2830416184-2834608127]: client csum 43a74539, server csum 5eb569c1 Mar 16 03:16:50 john25 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448280872016 length 1048576 too big: 1048092 left, 1048092 allowed Mar 16 03:16:50 arkle8 kernel: LustreError: Skipped 4 previous similar messages Mar 16 03:16:50 arkle8 kernel: LNet: Using FMR for registration Mar 16 03:16:50 arkle8 kernel: LNet: Skipped 1 previous similar message Mar 16 03:16:50 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882918bfc000 Mar 16 03:16:50 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882918bfc000 Mar 16 03:16:50 arkle8 kernel: LustreError: 108616:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88178af64050 x1594448285369712/t0(0) o4->219d9f55-42f0-6656-ab6f-f4de9ef8c6c7@192.168.44.127@o2ib44:443/0 lens 608/448 e 0 to 0 dl 1521130653 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:50 arkle8 kernel: Lustre: dagg-OST000f: Bulk IO write error with 219d9f55-42f0-6656-ab6f-f4de9ef8c6c7 (at 192.168.44.127@o2ib44), client will retry: rc = -110 Mar 16 03:16:50 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8821f16fae00 Mar 16 03:16:50 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8821f16fae00 Mar 16 03:16:50 john27 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448285369712 length 1048576 too big: 1046096 left, 1046096 allowed Mar 16 03:16:50 john28 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448282246752 length 1048576 too big: 1045344 left, 1045344 allowed Mar 16 03:16:50 john21 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448332911728 length 1048576 too big: 1044760 left, 1044760 allowed Mar 16 03:16:50 arkle7 kernel: LustreError: 168-f: dagg-OST000d: BAD WRITE CHECKSUM: from 12345-192.168.44.113@o2ib44 inode [0x200010636:0x44b:0x0] object 0x0:8477523 extent [2456919544-2461110271]: client csum 6dabae09, server csum 1cb00897 Mar 16 03:16:50 arkle7 kernel: LustreError: Skipped 5 previous similar messages Mar 16 03:16:50 john20 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448327525968 length 1048576 too big: 1045364 left, 1045364 allowed Mar 16 03:16:50 john15 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448323936064 length 1048576 too big: 1045944 left, 1045944 allowed Mar 16 03:16:50 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880710ec6a00 Mar 16 03:16:50 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880710ec6a00 Mar 16 03:16:50 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8821065a6c00 Mar 16 03:16:50 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8821065a6c00 Mar 16 03:16:50 arkle4 kernel: LNet: Using FMR for registration Mar 16 03:16:50 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88243c518400 Mar 16 03:16:50 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88243c518400 Mar 16 03:16:50 arkle4 kernel: LustreError: 302876:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882b0fae3c50 x1594448319400640/t0(0) o4->95b9add5-8117-706c-dad4-c3e4c830ffa8@192.168.44.107@o2ib44:443/0 lens 608/448 e 0 to 0 dl 1521130653 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:50 arkle4 kernel: Lustre: dagg-OST0006: Bulk IO write error with 95b9add5-8117-706c-dad4-c3e4c830ffa8 (at 192.168.44.107@o2ib44), client will retry: rc = -110 Mar 16 03:16:50 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880127ff5400 Mar 16 03:16:50 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880127ff5400 Mar 16 03:16:50 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8826a63dae00 Mar 16 03:16:50 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8826a63dae00 Mar 16 03:16:50 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8816d8095200 Mar 16 03:16:50 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8816d8095200 Mar 16 03:16:50 arkle4 kernel: LNet: Skipped 2 previous similar messages Mar 16 03:16:50 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8812ae54b800 Mar 16 03:16:50 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8812ae54b800 Mar 16 03:16:50 arkle7 kernel: LustreError: 34910:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8804bff11050 x1594448332914000/t0(0) o4->2c1132a9-8714-fc22-fa35-2c73e98cd45e@192.168.44.121@o2ib44:443/0 lens 608/448 e 0 to 0 dl 1521130653 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:50 john22 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448280685024 length 1048576 too big: 1048436 left, 1048436 allowed Mar 16 03:16:50 john30 kernel: LNetError: 897:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594699400481984 length 1048576 too big: 1048516 left, 1048516 allowed Mar 16 03:16:50 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e07be7200 Mar 16 03:16:50 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e07be7200 Mar 16 03:16:50 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8812ae54b000 Mar 16 03:16:50 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8812ae54b000 Mar 16 03:16:50 arkle7 kernel: Lustre: dagg-OST000c: Bulk IO write error with 23017e25-beb4-2d0c-1e7b-9a1bd7cbb415 (at 192.168.44.114@o2ib44), client will retry: rc = -110 Mar 16 03:16:50 arkle7 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:16:50 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882fca081e00 Mar 16 03:16:50 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882fca081e00 Mar 16 03:16:50 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881356e14600 Mar 16 03:16:50 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881356e14600 Mar 16 03:16:50 arkle7 kernel: LustreError: 34910:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 5 previous similar messages Mar 16 03:16:51 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882160282600 Mar 16 03:16:51 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882160282600 Mar 16 03:16:51 arkle4 kernel: LustreError: 168-f: dagg-OST0006: BAD WRITE CHECKSUM: from 12345-192.168.44.114@o2ib44 inode [0x68001c98b:0x47d:0x0] object 0x580000401:9544867 extent [2473939216-2478133247]: client csum 356741e6, server csum c8736695 Mar 16 03:16:51 arkle4 kernel: LustreError: Skipped 7 previous similar messages Mar 16 03:16:51 arkle8 kernel: LustreError: 168-f: dagg-OST000f: BAD WRITE CHECKSUM: from 12345-192.168.44.132@o2ib44 inode [0x68001c9cb:0x1d3:0x0] object 0x380000401:9525425 extent [2893565248-2897756159]: client csum b67f8c4d, server csum 1bf5d84 Mar 16 03:16:51 arkle8 kernel: LustreError: Skipped 10 previous similar messages Mar 16 03:16:51 arkle7 kernel: LustreError: 168-f: dagg-OST000c: BAD WRITE CHECKSUM: from 12345-192.168.44.116@o2ib44 inode [0x68001c985:0x44c:0x0] object 0x400000401:9549504 extent [2650216804-2654408703]: client csum 6fe100a1, server csum cb14988b Mar 16 03:16:51 arkle7 kernel: LustreError: Skipped 13 previous similar messages Mar 16 03:16:51 arkle5 kernel: LustreError: 168-f: dagg-OST0009: BAD WRITE CHECKSUM: from 12345-192.168.44.116@o2ib44 inode [0x68001c985:0x44e:0x0] object 0x340000401:9549664 extent [2394871372-2399064063]: client csum e9a8b437, server csum 60907f58 Mar 16 03:16:51 arkle5 kernel: LustreError: Skipped 2 previous similar messages Mar 16 03:16:51 arkle6 kernel: LustreError: 168-f: dagg-OST000a: BAD WRITE CHECKSUM: from 12345-192.168.44.109@o2ib44 inode [0x200010631:0x3b9:0x0] object 0x0:8426505 extent [2540803804-2544996351]: client csum a37f43ee, server csum b2099d0e Mar 16 03:16:51 john13 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594448319196096 length 1048576 too big: 1046240 left, 1046240 allowed Mar 16 03:16:51 john19 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448327426912 length 1048576 too big: 1048520 left, 1048520 allowed Mar 16 03:16:51 john33 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594699393878992 length 1048576 too big: 1046348 left, 1046348 allowed Mar 16 03:16:51 john32 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594699396460784 length 1048576 too big: 1046448 left, 1046448 allowed Mar 16 03:16:51 john26 kernel: LNetError: 904:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594448281315440 length 1048576 too big: 1046228 left, 1046228 allowed Mar 16 03:16:51 john12 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448324464560 length 1048576 too big: 1047560 left, 1047560 allowed Mar 16 03:16:51 arkle6 kernel: LNet: Using FMR for registration Mar 16 03:16:51 john30 kernel: LNetError: 897:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594699400484576 length 1048576 too big: 1045636 left, 1045636 allowed Mar 16 03:16:51 john30 kernel: LNetError: 897:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:51 john21 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448332914704 length 1048576 too big: 1047916 left, 1047916 allowed Mar 16 03:16:51 john21 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:51 john7 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594448319403952 length 1048576 too big: 1047924 left, 1047924 allowed Mar 16 03:16:51 john9 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448323945264 length 1048576 too big: 1044696 left, 1044696 allowed Mar 16 03:16:51 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8826a63df200 Mar 16 03:16:51 john20 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448327528752 length 1048576 too big: 1047744 left, 1047744 allowed Mar 16 03:16:51 john16 kernel: LNetError: 912:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448321485008 length 1048576 too big: 1046104 left, 1046104 allowed Mar 16 03:16:51 john29 kernel: LNetError: 903:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594699419066272 length 1048576 too big: 1045736 left, 1045736 allowed Mar 16 03:16:51 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ed05ee000 Mar 16 03:16:51 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ed05ee000 Mar 16 03:16:51 arkle6 kernel: LNet: Skipped 1 previous similar message Mar 16 03:16:51 arkle6 kernel: LustreError: 277273:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8818d9519450 x1594699393878992/t0(0) o4->22c507bc-bb1f-dbcf-8316-529c1039bbfe@192.168.44.133@o2ib44:444/0 lens 608/448 e 0 to 0 dl 1521130654 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:51 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dca55f600 Mar 16 03:16:51 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dca55f600 Mar 16 03:16:51 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c3cd8d600 Mar 16 03:16:51 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c3cd8d600 Mar 16 03:16:51 arkle6 kernel: Lustre: dagg-OST000a: Bulk IO write error with d4acd478-18fd-71d3-c177-089c607f8639 (at 192.168.44.119@o2ib44), client will retry: rc = -110 Mar 16 03:16:51 arkle6 kernel: LustreError: 277273:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 2 previous similar messages Mar 16 03:16:51 arkle5 kernel: LNet: Using FMR for registration Mar 16 03:16:51 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817dd2f4200 Mar 16 03:16:51 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817dd2f4200 Mar 16 03:16:51 arkle5 kernel: LustreError: 171334:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881619b02850 x1594448319196096/t0(0) o4->240b2d78-e95a-c921-2bc6-0bc7648161c4@192.168.44.113@o2ib44:444/0 lens 608/448 e 0 to 0 dl 1521130654 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:51 arkle5 kernel: Lustre: dagg-OST0009: Bulk IO write error with 240b2d78-e95a-c921-2bc6-0bc7648161c4 (at 192.168.44.113@o2ib44), client will retry: rc = -110 Mar 16 03:16:51 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f2aaaa200 Mar 16 03:16:51 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f2aaaa200 Mar 16 03:16:51 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881e4b6e8800 Mar 16 03:16:51 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881e4b6e8800 Mar 16 03:16:51 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880802fdbe00 Mar 16 03:16:51 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880802fdbe00 Mar 16 03:16:51 arkle5 kernel: LNet: Skipped 2 previous similar messages Mar 16 03:16:51 john15 kernel: LNetError: 904:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594448323938480 length 1048576 too big: 1046772 left, 1046772 allowed Mar 16 03:16:51 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8826a63df200 Mar 16 03:16:51 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881581857400 Mar 16 03:16:51 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881581857400 Mar 16 03:16:51 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8c400 Mar 16 03:16:51 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8c400 Mar 16 03:16:51 arkle4 kernel: LustreError: 176417:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8816d8c67450 x1594448321485008/t0(0) o4->555ab62f-a8a1-c567-5cc2-8766e0a20f71@192.168.44.116@o2ib44:444/0 lens 608/448 e 0 to 0 dl 1521130654 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:51 arkle4 kernel: LustreError: 176417:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 4 previous similar messages Mar 16 03:16:51 arkle4 kernel: Lustre: dagg-OST0007: Bulk IO write error with 555ab62f-a8a1-c567-5cc2-8766e0a20f71 (at 192.168.44.116@o2ib44), client will retry: rc = -110 Mar 16 03:16:51 arkle4 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:16:51 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8815a47f0800 Mar 16 03:16:51 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8815a47f0800 Mar 16 03:16:51 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c7ffa8e00 Mar 16 03:16:51 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c7ffa8e00 Mar 16 03:16:51 john8 kernel: LNetError: 914:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448316803760 length 1048576 too big: 1047284 left, 1047284 allowed Mar 16 03:16:51 arkle3 kernel: LustreError: 168-f: dagg-OST0004: BAD WRITE CHECKSUM: from 12345-192.168.44.121@o2ib44 inode [0x28001c1f6:0x262:0x0] object 0x540000400:8140299 extent [2511758908-2515951615]: client csum 726a9e4a, server csum f244d308 Mar 16 03:16:51 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882a84f8c200 Mar 16 03:16:51 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882a84f8c200 Mar 16 03:16:51 arkle3 kernel: LustreError: 131836:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882f75ae2850 x1594699400484576/t0(0) o4->faed83b4-3a5e-2a8e-2882-7e99d174dfb0@192.168.44.130@o2ib44:444/0 lens 608/448 e 0 to 0 dl 1521130654 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:51 arkle3 kernel: Lustre: dagg-OST0004: Bulk IO write error with faed83b4-3a5e-2a8e-2882-7e99d174dfb0 (at 192.168.44.130@o2ib44), client will retry: rc = -110 Mar 16 03:16:51 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881347069800 Mar 16 03:16:51 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881347069800 Mar 16 03:16:51 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882474793600 Mar 16 03:16:51 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882474793600 Mar 16 03:16:51 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8815febd6a00 Mar 16 03:16:51 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8815febd6a00 Mar 16 03:16:51 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880ef73d3800 Mar 16 03:16:51 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880ef73d3800 Mar 16 03:16:51 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880ef73d3800 Mar 16 03:16:51 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880ef73d3800 Mar 16 03:16:51 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c6c9ad000 Mar 16 03:16:51 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c6c9ad000 Mar 16 03:16:51 arkle3 kernel: LustreError: Skipped 7 previous similar messages Mar 16 03:16:51 john27 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448285372496 length 1048576 too big: 1048252 left, 1048252 allowed Mar 16 03:16:51 john6 kernel: LNetError: 914:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448323927280 length 1048576 too big: 1047972 left, 1047972 allowed Mar 16 03:16:51 john24 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448284017648 length 1048576 too big: 1044708 left, 1044708 allowed Mar 16 03:16:51 john18 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448284367264 length 1048576 too big: 1048556 left, 1048556 allowed Mar 16 03:16:51 john22 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448280687712 length 1048576 too big: 1045244 left, 1045244 allowed Mar 16 03:16:51 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88156ae50c00 Mar 16 03:16:51 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88156ae50c00 Mar 16 03:16:51 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881596f90800 Mar 16 03:16:51 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881596f90800 Mar 16 03:16:51 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8805ddfaec00 Mar 16 03:16:51 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8805ddfaec00 Mar 16 03:16:51 arkle1 kernel: LustreError: 168-f: dagg-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.44.113@o2ib44 inode [0x200010636:0x44c:0x0] object 0x0:8441064 extent [2609978956-2614169599]: client csum a795acda, server csum 24d58d70 Mar 16 03:16:51 arkle1 kernel: LNet: Using FMR for registration Mar 16 03:16:51 arkle1 kernel: LNet: Skipped 1 previous similar message Mar 16 03:16:51 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881577282400 Mar 16 03:16:51 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881577282400 Mar 16 03:16:51 arkle1 kernel: LustreError: 277918:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8817dbcf5850 x1594448316803760/t0(0) o4->15677bdd-7206-6521-60ab-6c4f2bdd542b@192.168.44.108@o2ib44:444/0 lens 608/448 e 0 to 0 dl 1521130654 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:51 arkle1 kernel: Lustre: dagg-OST0000: Bulk IO write error with 15677bdd-7206-6521-60ab-6c4f2bdd542b (at 192.168.44.108@o2ib44), client will retry: rc = -110 Mar 16 03:16:51 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882744646400 Mar 16 03:16:51 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882744646400 Mar 16 03:16:51 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ee3b4dc00 Mar 16 03:16:51 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ee3b4dc00 Mar 16 03:16:51 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8828a1579400 Mar 16 03:16:51 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8828a1579400 Mar 16 03:16:51 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8822ef652e00 Mar 16 03:16:51 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8822ef652e00 Mar 16 03:16:51 arkle1 kernel: LustreError: Skipped 1 previous similar message Mar 16 03:16:51 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e7e31bc00 Mar 16 03:16:51 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e7e31bc00 Mar 16 03:16:51 arkle2 kernel: LustreError: 168-f: dagg-OST0002: BAD WRITE CHECKSUM: from 12345-192.168.44.128@o2ib44 inode [0x28001c1f5:0x268:0x0] object 0x4c0000400:8144265 extent [2694503584-2698694655]: client csum c0f7587f, server csum c1247393 Mar 16 03:16:51 arkle2 kernel: LNet: Using FMR for registration Mar 16 03:16:51 arkle2 kernel: LNet: Skipped 1 previous similar message Mar 16 03:16:51 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8821a4f15200 Mar 16 03:16:51 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8821a4f15200 Mar 16 03:16:51 arkle2 kernel: LustreError: Skipped 1 previous similar message Mar 16 03:16:51 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881042650800 Mar 16 03:16:51 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881042650800 Mar 16 03:16:51 john23 kernel: LNetError: 914:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448282171968 length 1048576 too big: 1047668 left, 1047668 allowed Mar 16 03:16:51 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88156ae51e00 Mar 16 03:16:51 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88156ae51e00 Mar 16 03:16:51 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881042653000 Mar 16 03:16:51 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881042653000 Mar 16 03:16:51 arkle2 kernel: LustreError: 349954:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88176f6dbc50 x1594699419068752/t0(0) o4->c2000ae7-b551-1f4a-4f30-41e5a9e31fb7@192.168.44.129@o2ib44:444/0 lens 608/448 e 0 to 0 dl 1521130654 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:51 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881a10752400 Mar 16 03:16:51 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881a10752400 Mar 16 03:16:51 arkle2 kernel: Lustre: dagg-OST0002: Bulk IO write error with e5e03fa1-aa9e-78b9-d06c-6e18b34089cf (at 192.168.44.123@o2ib44), client will retry: rc = -110 Mar 16 03:16:51 arkle2 kernel: LustreError: 349954:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 1 previous similar message Mar 16 03:16:51 john12 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448324467632 length 1048576 too big: 1045492 left, 1045492 allowed Mar 16 03:16:51 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88279a5d7c00 Mar 16 03:16:51 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88279a5d7c00 Mar 16 03:16:51 arkle3 kernel: LustreError: 120216:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882f75ae0050 x1594448324467632/t0(0) o4->2dbd7de7-b12b-3af0-6d22-757cd258d9d0@192.168.44.112@o2ib44:444/0 lens 608/448 e 0 to 0 dl 1521130654 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:51 john17 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448316853392 length 1048576 too big: 1044508 left, 1044508 allowed Mar 16 03:16:51 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882a33d0ae00 Mar 16 03:16:51 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882a33d0ae00 Mar 16 03:16:51 arkle3 kernel: LustreError: 168-f: dagg-OST0005: BAD WRITE CHECKSUM: from 12345-192.168.44.130@o2ib44 inode [0x68001c9d4:0x1f2:0x0] object 0x2c0000401:9597153 extent [2723997364-2728189951]: client csum dd9d3ea9, server csum 9a6b38b9 Mar 16 03:16:51 arkle3 kernel: LustreError: Skipped 9 previous similar messages Mar 16 03:16:51 arkle3 kernel: LustreError: 120216:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 6 previous similar messages Mar 16 03:16:51 arkle5 kernel: LustreError: 168-f: dagg-OST0009: BAD WRITE CHECKSUM: from 12345-192.168.44.125@o2ib44 inode [0x68001c9c4:0x26f:0x0] object 0x340000401:9549662 extent [2203093084-2207285247]: client csum 40bf4003, server csum 719f0a1 Mar 16 03:16:51 arkle5 kernel: LustreError: Skipped 11 previous similar messages Mar 16 03:16:51 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881e4b6ed200 Mar 16 03:16:51 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881e4b6ed200 Mar 16 03:16:51 arkle5 kernel: LustreError: 329999:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff880159ef4850 x1594448332915424/t0(0) o4->2c1132a9-8714-fc22-fa35-2c73e98cd45e@192.168.44.121@o2ib44:444/0 lens 608/448 e 0 to 0 dl 1521130654 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:51 arkle5 kernel: LustreError: 329999:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 6 previous similar messages Mar 16 03:16:51 arkle5 kernel: Lustre: dagg-OST0008: Bulk IO write error with 2c1132a9-8714-fc22-fa35-2c73e98cd45e (at 192.168.44.121@o2ib44), client will retry: rc = -110 Mar 16 03:16:51 arkle5 kernel: Lustre: Skipped 6 previous similar messages Mar 16 03:16:51 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882bbb349800 Mar 16 03:16:51 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882bbb349800 Mar 16 03:16:51 john31 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594699394166672 length 1048576 too big: 1048064 left, 1048064 allowed Mar 16 03:16:52 arkle3 kernel: Lustre: dagg-OST0005: Bulk IO write error with 2dbd7de7-b12b-3af0-6d22-757cd258d9d0 (at 192.168.44.112@o2ib44), client will retry: rc = -110 Mar 16 03:16:52 arkle3 kernel: Lustre: Skipped 6 previous similar messages Mar 16 03:16:52 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ecf6fe000 Mar 16 03:16:52 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ecf6fe000 Mar 16 03:16:52 john25 kernel: LNetError: 903:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448280876144 length 1048576 too big: 1045400 left, 1045400 allowed Mar 16 03:16:52 john25 kernel: LNetError: 903:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:52 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8816f4865800 Mar 16 03:16:52 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8816f4865800 Mar 16 03:16:52 arkle1 kernel: LustreError: 168-f: dagg-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.44.120@o2ib44 inode [0x68001c9c6:0x29c:0x0] object 0x240000401:9555677 extent [513722540-517914623]: client csum 9ea3f720, server csum adb3c17 Mar 16 03:16:52 arkle1 kernel: LustreError: Skipped 6 previous similar messages Mar 16 03:16:52 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882a33d0dc00 Mar 16 03:16:52 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882a33d0dc00 Mar 16 03:16:52 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8816f4867c00 Mar 16 03:16:52 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8816f4867c00 Mar 16 03:16:52 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8802ce99e400 Mar 16 03:16:52 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8802ce99e400 Mar 16 03:16:52 arkle4 kernel: LustreError: 168-f: dagg-OST0007: BAD WRITE CHECKSUM: from 12345-192.168.44.133@o2ib44 inode [0x68001c9d3:0x1fb:0x0] object 0x300000401:9577187 extent [2794923880-2799116287]: client csum 30ecc4dc, server csum 9ad54bb2 Mar 16 03:16:52 arkle4 kernel: LustreError: Skipped 22 previous similar messages Mar 16 03:16:52 john33 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594699393883264 length 1048576 too big: 1047376 left, 1047376 allowed Mar 16 03:16:52 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881312b1e200 Mar 16 03:16:52 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881312b1e200 Mar 16 03:16:52 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817c7a2da00 Mar 16 03:16:52 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817c7a2da00 Mar 16 03:16:52 arkle7 kernel: LustreError: 168-f: dagg-OST000d: BAD WRITE CHECKSUM: from 12345-192.168.44.116@o2ib44 inode [0x28001c1be:0x471:0x0] object 0x440000400:8183010 extent [2838840976-2843033599]: client csum 33ff46ec, server csum 9b63c610 Mar 16 03:16:52 arkle7 kernel: LustreError: Skipped 25 previous similar messages Mar 16 03:16:52 arkle1 kernel: LustreError: 395413:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882f3e2b3450 x1594448282176160/t0(0) o4->e5e03fa1-aa9e-78b9-d06c-6e18b34089cf@192.168.44.123@o2ib44:445/0 lens 608/448 e 0 to 0 dl 1521130655 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:52 arkle1 kernel: LustreError: 395413:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 4 previous similar messages Mar 16 03:16:52 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8829bd9d7800 Mar 16 03:16:52 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8829bd9d7800 Mar 16 03:16:52 john6 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448323930016 length 1048576 too big: 1044832 left, 1044832 allowed Mar 16 03:16:52 arkle1 kernel: Lustre: dagg-OST0001: Bulk IO write error with e5e03fa1-aa9e-78b9-d06c-6e18b34089cf (at 192.168.44.123@o2ib44), client will retry: rc = -110 Mar 16 03:16:52 arkle1 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:16:52 arkle8 kernel: LustreError: 168-f: dagg-OST000f: BAD WRITE CHECKSUM: from 12345-192.168.44.110@o2ib44 inode [0x200010630:0x432:0x0] object 0x0:8441565 extent [2782907272-2787098623]: client csum 68e89ac9, server csum dff25044 Mar 16 03:16:52 arkle8 kernel: LustreError: Skipped 9 previous similar messages Mar 16 03:16:52 arkle6 kernel: LustreError: 168-f: dagg-OST000b: BAD WRITE CHECKSUM: from 12345-192.168.44.128@o2ib44 inode [0x28001c1f5:0x265:0x0] object 0x3c0000400:8148560 extent [2314777612-2318970879]: client csum 13785fcc, server csum 6d80753d Mar 16 03:16:52 arkle6 kernel: LustreError: Skipped 18 previous similar messages Mar 16 03:16:52 john7 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448319405744 length 1048576 too big: 1048392 left, 1048392 allowed Mar 16 03:16:52 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8816b89b8400 Mar 16 03:16:52 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8816b89b8400 Mar 16 03:16:52 arkle2 kernel: LustreError: 238714:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff880d04250c50 x1594448323930016/t0(0) o4->efbad07f-625e-4fc5-c94b-ef8442fed4fb@192.168.44.106@o2ib44:445/0 lens 608/448 e 0 to 0 dl 1521130655 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:52 arkle2 kernel: LustreError: 238714:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 5 previous similar messages Mar 16 03:16:52 arkle2 kernel: Lustre: dagg-OST0002: Bulk IO write error with efbad07f-625e-4fc5-c94b-ef8442fed4fb (at 192.168.44.106@o2ib44), client will retry: rc = -110 Mar 16 03:16:52 arkle2 kernel: Lustre: Skipped 6 previous similar messages Mar 16 03:16:52 arkle2 kernel: LustreError: 168-f: dagg-OST0003: BAD WRITE CHECKSUM: from 12345-192.168.44.128@o2ib44 inode [0x28001c1f5:0x264:0x0] object 0x500000400:8117103 extent [2671251064-2675441663]: client csum 31582de0, server csum a327e9d9 Mar 16 03:16:52 arkle2 kernel: LustreError: Skipped 3 previous similar messages Mar 16 03:16:52 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882efa474600 Mar 16 03:16:52 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882efa474600 Mar 16 03:16:52 john9 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448323950288 length 1048576 too big: 1046800 left, 1046800 allowed Mar 16 03:16:52 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882239a55200 Mar 16 03:16:52 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882239a55200 Mar 16 03:16:52 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c6c9ad400 Mar 16 03:16:52 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c6c9ad400 Mar 16 03:16:52 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ecf6fcc00 Mar 16 03:16:52 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ecf6fcc00 Mar 16 03:16:52 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d73db7200 Mar 16 03:16:52 john28 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448282252368 length 1048576 too big: 1046868 left, 1046868 allowed Mar 16 03:16:52 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18ca200 Mar 16 03:16:52 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18ca200 Mar 16 03:16:52 john20 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448327532816 length 1048576 too big: 1047508 left, 1047508 allowed Mar 16 03:16:52 john20 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:52 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d73db7200 Mar 16 03:16:52 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8821a4f14c00 Mar 16 03:16:52 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f49d75c00 Mar 16 03:16:52 john8 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448316806480 length 1048576 too big: 1047612 left, 1047612 allowed Mar 16 03:16:52 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882393c2d400 Mar 16 03:16:52 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882393c2d400 Mar 16 03:16:52 john32 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594699396465568 length 1048576 too big: 1048468 left, 1048468 allowed Mar 16 03:16:52 john32 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:52 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f49d75c00 Mar 16 03:16:52 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8821a4f14c00 Mar 16 03:16:52 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881794bafc00 Mar 16 03:16:52 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881794bafc00 Mar 16 03:16:52 arkle4 kernel: LustreError: 176442:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8809288c4c50 x1594699396465568/t0(0) o4->0202600d-b066-c547-a639-ee53deb97148@192.168.44.132@o2ib44:445/0 lens 608/448 e 0 to 0 dl 1521130655 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:52 john29 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594699419069488 length 1048576 too big: 1047832 left, 1047832 allowed Mar 16 03:16:52 john29 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 2 previous similar messages Mar 16 03:16:52 arkle4 kernel: LustreError: 176442:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 5 previous similar messages Mar 16 03:16:52 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c7ffae200 Mar 16 03:16:52 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c7ffae200 Mar 16 03:16:52 arkle7 kernel: LustreError: 189392:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881766690450 x1594699419069488/t0(0) o4->c2000ae7-b551-1f4a-4f30-41e5a9e31fb7@192.168.44.129@o2ib44:445/0 lens 608/448 e 0 to 0 dl 1521130655 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:52 arkle4 kernel: Lustre: dagg-OST0006: Bulk IO write error with 0202600d-b066-c547-a639-ee53deb97148 (at 192.168.44.132@o2ib44), client will retry: rc = -110 Mar 16 03:16:52 arkle4 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:16:52 arkle7 kernel: LustreError: 189392:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 2 previous similar messages Mar 16 03:16:52 arkle7 kernel: Lustre: dagg-OST000d: Bulk IO write error with c2000ae7-b551-1f4a-4f30-41e5a9e31fb7 (at 192.168.44.129@o2ib44), client will retry: rc = -110 Mar 16 03:16:52 arkle7 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:16:52 john23 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594448282177696 length 1048576 too big: 1046948 left, 1046948 allowed Mar 16 03:16:52 john23 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:52 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8815a47f2c00 Mar 16 03:16:52 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8815a47f2c00 Mar 16 03:16:52 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ed05ef200 Mar 16 03:16:52 john19 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448327431632 length 1048576 too big: 1047928 left, 1047928 allowed Mar 16 03:16:52 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ed05ef200 Mar 16 03:16:52 arkle6 kernel: LustreError: 149762:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8818d9519850 x1594448323951216/t0(0) o4->7d6768d1-f792-7c54-7167-7223c4aae84c@192.168.44.109@o2ib44:445/0 lens 608/448 e 0 to 0 dl 1521130655 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:52 arkle6 kernel: LustreError: 149762:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 2 previous similar messages Mar 16 03:16:52 arkle6 kernel: Lustre: dagg-OST000b: Bulk IO write error with 7d6768d1-f792-7c54-7167-7223c4aae84c (at 192.168.44.109@o2ib44), client will retry: rc = -110 Mar 16 03:16:52 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177695c200 Mar 16 03:16:52 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177695c200 Mar 16 03:16:52 arkle6 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:16:53 arkle3 kernel: LustreError: 168-f: dagg-OST0005: BAD WRITE CHECKSUM: from 12345-192.168.44.109@o2ib44 inode [0x68001c982:0x487:0x0] object 0x2c0000401:9597154 extent [2875017568-2879209471]: client csum df657bb8, server csum aa9859db Mar 16 03:16:53 arkle3 kernel: LustreError: Skipped 19 previous similar messages Mar 16 03:16:53 john31 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594699394169536 length 1048576 too big: 1046072 left, 1046072 allowed Mar 16 03:16:53 john31 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:53 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882367f9a400 Mar 16 03:16:53 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882367f9a400 Mar 16 03:16:53 arkle5 kernel: LustreError: 168-f: dagg-OST0009: BAD WRITE CHECKSUM: from 12345-192.168.44.106@o2ib44 inode [0x68001c980:0x460:0x0] object 0x340000401:9549663 extent [2679641824-2683834367]: client csum 8f73ff5a, server csum 65d496a1 Mar 16 03:16:53 arkle5 kernel: LustreError: Skipped 14 previous similar messages Mar 16 03:16:53 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881775759000 Mar 16 03:16:53 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881775759000 Mar 16 03:16:53 arkle1 kernel: LustreError: 168-f: dagg-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.44.126@o2ib44 inode [0x28001c1fa:0x246:0x0] object 0x480000400:8167474 extent [2845350400-2849542143]: client csum b209d5c8, server csum 8567dd04 Mar 16 03:16:53 arkle1 kernel: LustreError: Skipped 40 previous similar messages Mar 16 03:16:53 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8820dec22600 Mar 16 03:16:53 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8820dec22600 Mar 16 03:16:53 john10 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448319882256 length 1048576 too big: 1046492 left, 1046492 allowed Mar 16 03:16:53 john18 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448284373008 length 1048576 too big: 1045680 left, 1045680 allowed Mar 16 03:16:53 john18 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:53 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8828c7e3be00 Mar 16 03:16:53 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8828c7e3be00 Mar 16 03:16:53 john16 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448321508048 length 1048576 too big: 1047312 left, 1047312 allowed Mar 16 03:16:53 john16 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:53 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e31e7b800 Mar 16 03:16:53 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e31e7b800 Mar 16 03:16:53 john12 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448324468928 length 1048576 too big: 1047156 left, 1047156 allowed Mar 16 03:16:53 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881d6d5d9400 Mar 16 03:16:53 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881d6d5d9400 Mar 16 03:16:53 arkle2 kernel: LustreError: 168-f: dagg-OST0003: BAD WRITE CHECKSUM: from 12345-192.168.44.117@o2ib44 inode [0x28001c1bb:0x4b0:0x0] object 0x500000400:8117105 extent [2561975236-2566168575]: client csum c603a02a, server csum 70b23ceb Mar 16 03:16:53 arkle2 kernel: LustreError: Skipped 13 previous similar messages Mar 16 03:16:53 john33 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594699393884960 length 1048576 too big: 1047948 left, 1047948 allowed Mar 16 03:16:53 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8826081b3400 Mar 16 03:16:53 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8826081b3400 Mar 16 03:16:53 arkle8 kernel: LustreError: 48894:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882bb9dd1050 x1594699393884960/t0(0) o4->22c507bc-bb1f-dbcf-8316-529c1039bbfe@192.168.44.133@o2ib44:446/0 lens 608/448 e 0 to 0 dl 1521130656 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:53 arkle8 kernel: LustreError: 48894:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 2 previous similar messages Mar 16 03:16:53 arkle8 kernel: Lustre: dagg-OST000f: Bulk IO write error with 22c507bc-bb1f-dbcf-8316-529c1039bbfe (at 192.168.44.133@o2ib44), client will retry: rc = -110 Mar 16 03:16:53 arkle8 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:16:53 john14 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448317087008 length 1048576 too big: 1045352 left, 1045352 allowed Mar 16 03:16:53 john14 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:53 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808e90ea000 Mar 16 03:16:53 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808e90ea000 Mar 16 03:16:53 john30 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594699400487456 length 1048576 too big: 1044848 left, 1044848 allowed Mar 16 03:16:53 john30 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:53 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88252e424200 Mar 16 03:16:53 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88252e424200 Mar 16 03:16:53 arkle3 kernel: LustreError: 94805:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882ed17d0450 x1594699400487456/t0(0) o4->faed83b4-3a5e-2a8e-2882-7e99d174dfb0@192.168.44.130@o2ib44:446/0 lens 608/448 e 0 to 0 dl 1521130656 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:53 arkle3 kernel: LustreError: 94805:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 1 previous similar message Mar 16 03:16:53 arkle3 kernel: Lustre: dagg-OST0005: Bulk IO write error with faed83b4-3a5e-2a8e-2882-7e99d174dfb0 (at 192.168.44.130@o2ib44), client will retry: rc = -110 Mar 16 03:16:53 arkle3 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:16:53 john13 kernel: LNetError: 912:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448319201264 length 1048576 too big: 1048356 left, 1048356 allowed Mar 16 03:16:53 john13 kernel: LNetError: 912:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:16:53 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ee3b4bc00 Mar 16 03:16:53 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ee3b4bc00 Mar 16 03:16:53 arkle1 kernel: LustreError: 394807:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88175c7f8050 x1594448319201264/t0(0) o4->240b2d78-e95a-c921-2bc6-0bc7648161c4@192.168.44.113@o2ib44:446/0 lens 608/448 e 0 to 0 dl 1521130656 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:53 john10 kernel: LNetError: 912:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448319886272 length 1048576 too big: 1046464 left, 1046464 allowed Mar 16 03:16:53 arkle1 kernel: LustreError: 394807:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 5 previous similar messages Mar 16 03:16:54 arkle1 kernel: Lustre: dagg-OST0000: Bulk IO write error with 240b2d78-e95a-c921-2bc6-0bc7648161c4 (at 192.168.44.113@o2ib44), client will retry: rc = -110 Mar 16 03:16:54 arkle1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:16:54 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88293fbcae00 Mar 16 03:16:54 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88293fbcae00 Mar 16 03:16:54 john32 kernel: LNetError: 897:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594699396467776 length 1048576 too big: 1047136 left, 1047136 allowed Mar 16 03:16:54 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8822aef94200 Mar 16 03:16:54 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8822aef94200 Mar 16 03:16:54 john6 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448323933968 length 1048576 too big: 1047580 left, 1047580 allowed Mar 16 03:16:54 john6 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:54 arkle4 kernel: LustreError: 168-f: dagg-OST0007: BAD WRITE CHECKSUM: from 12345-192.168.44.132@o2ib44 inode [0x28001c206:0x239:0x0] object 0x300000400:8185286 extent [2719794400-2723987455]: client csum ea9b6e0a, server csum 95766137 Mar 16 03:16:54 arkle4 kernel: LustreError: Skipped 26 previous similar messages Mar 16 03:16:54 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8829271bca00 Mar 16 03:16:54 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8829271bca00 Mar 16 03:16:54 arkle6 kernel: LustreError: 168-f: dagg-OST000a: BAD WRITE CHECKSUM: from 12345-192.168.44.109@o2ib44 inode [0x200010631:0x3b9:0x0] object 0x0:8426505 extent [2850004924-2853693627]: client csum b777c225, server csum 7aabdefa Mar 16 03:16:54 arkle6 kernel: LustreError: Skipped 54 previous similar messages Mar 16 03:16:54 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882722f96a00 Mar 16 03:16:54 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882722f96a00 Mar 16 03:16:54 arkle6 kernel: LustreError: 330065:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88094712e850 x1594448317088528/t0(0) o4->23017e25-beb4-2d0c-1e7b-9a1bd7cbb415@192.168.44.114@o2ib44:447/0 lens 608/448 e 0 to 0 dl 1521130657 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:54 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177575c200 Mar 16 03:16:54 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177575c200 Mar 16 03:16:54 arkle6 kernel: LustreError: 330065:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 2 previous similar messages Mar 16 03:16:54 arkle6 kernel: Lustre: dagg-OST000b: Bulk IO write error with 23017e25-beb4-2d0c-1e7b-9a1bd7cbb415 (at 192.168.44.114@o2ib44), client will retry: rc = -110 Mar 16 03:16:54 arkle6 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:16:54 arkle7 kernel: LustreError: 168-f: dagg-OST000d: BAD WRITE CHECKSUM: from 12345-192.168.44.118@o2ib44 inode [0x20001066a:0x729:0x0] object 0x0:8477522 extent [2770097464-2774290431]: client csum 4b007140, server csum f26a0b93 Mar 16 03:16:54 arkle7 kernel: LustreError: Skipped 32 previous similar messages Mar 16 03:16:54 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881852ebe400 Mar 16 03:16:54 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881852ebe400 Mar 16 03:16:54 john28 kernel: LNetError: 904:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448282256576 length 1048576 too big: 1046696 left, 1046696 allowed Mar 16 03:16:54 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e3de8b400 Mar 16 03:16:54 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e3de8b400 Mar 16 03:16:54 arkle8 kernel: LustreError: 168-f: dagg-OST000f: BAD WRITE CHECKSUM: from 12345-192.168.44.108@o2ib44 inode [0x20001063a:0x4c2:0x0] object 0x0:8441566 extent [2994603028-2998796287]: client csum e1966ee3, server csum 28d1376a Mar 16 03:16:54 arkle8 kernel: LustreError: Skipped 21 previous similar messages Mar 16 03:16:54 john26 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448281333600 length 1048576 too big: 1045700 left, 1045700 allowed Mar 16 03:16:54 john26 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:54 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881577282400 Mar 16 03:16:54 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881577282400 Mar 16 03:16:54 john8 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448316809648 length 1048576 too big: 1046328 left, 1046328 allowed Mar 16 03:16:54 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881fb155ac00 Mar 16 03:16:54 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881fb155ac00 Mar 16 03:16:54 arkle8 kernel: LustreError: 178259:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881787165c50 x1594448316809648/t0(0) o4->15677bdd-7206-6521-60ab-6c4f2bdd542b@192.168.44.108@o2ib44:447/0 lens 608/448 e 0 to 0 dl 1521130657 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:55 arkle8 kernel: LustreError: 178259:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 2 previous similar messages Mar 16 03:16:55 arkle8 kernel: Lustre: dagg-OST000f: Bulk IO write error with 15677bdd-7206-6521-60ab-6c4f2bdd542b (at 192.168.44.108@o2ib44), client will retry: rc = -110 Mar 16 03:16:55 arkle8 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:16:55 arkle3 kernel: LustreError: 168-f: dagg-OST0004: BAD WRITE CHECKSUM: from 12345-192.168.44.110@o2ib44 inode [0x68001c981:0x4cd:0x0] object 0x540000401:9539582 extent [3243623416-3247816703]: client csum 7fda76e0, server csum 410da725 Mar 16 03:16:55 arkle3 kernel: LustreError: Skipped 33 previous similar messages Mar 16 03:16:55 arkle1 kernel: LustreError: 168-f: dagg-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.44.118@o2ib44 inode [0x68001c9bd:0x2a8:0x0] object 0x240000401:9555675 extent [2136348124-2140540927]: client csum 22d4067f, server csum dcdc276e Mar 16 03:16:55 arkle1 kernel: LustreError: Skipped 35 previous similar messages Mar 16 03:16:55 arkle5 kernel: LustreError: 168-f: dagg-OST0009: BAD WRITE CHECKSUM: from 12345-192.168.44.106@o2ib44 inode [0x68001c980:0x460:0x0] object 0x340000401:9549663 extent [2868385504-2872578047]: client csum a4850890, server csum d85a92af Mar 16 03:16:55 arkle5 kernel: LustreError: Skipped 25 previous similar messages Mar 16 03:16:55 john18 kernel: LNetError: 914:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448284393072 length 1048576 too big: 1046156 left, 1046156 allowed Mar 16 03:16:55 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88169cc5f800 Mar 16 03:16:55 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88169cc5f800 Mar 16 03:16:55 arkle5 kernel: LustreError: 164880:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8817c33d0850 x1594448323934928/t0(0) o4->efbad07f-625e-4fc5-c94b-ef8442fed4fb@192.168.44.106@o2ib44:448/0 lens 608/448 e 0 to 0 dl 1521130658 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:55 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac956600 Mar 16 03:16:55 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac956600 Mar 16 03:16:55 arkle5 kernel: LustreError: 164880:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 4 previous similar messages Mar 16 03:16:55 arkle5 kernel: Lustre: dagg-OST0009: Bulk IO write error with efbad07f-625e-4fc5-c94b-ef8442fed4fb (at 192.168.44.106@o2ib44), client will retry: rc = -110 Mar 16 03:16:55 arkle5 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:16:55 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88176627c800 Mar 16 03:16:55 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88176627c800 Mar 16 03:16:55 arkle2 kernel: LustreError: 350062:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff880c67647050 x1594448316810400/t0(0) o4->15677bdd-7206-6521-60ab-6c4f2bdd542b@192.168.44.108@o2ib44:448/0 lens 608/448 e 0 to 0 dl 1521130658 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:55 arkle2 kernel: LustreError: 350062:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 5 previous similar messages Mar 16 03:16:55 arkle2 kernel: Lustre: dagg-OST0003: Bulk IO write error with 15677bdd-7206-6521-60ab-6c4f2bdd542b (at 192.168.44.108@o2ib44), client will retry: rc = -110 Mar 16 03:16:55 arkle2 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:16:55 arkle2 kernel: LustreError: 168-f: dagg-OST0003: BAD WRITE CHECKSUM: from 12345-192.168.44.117@o2ib44 inode [0x28001c1bb:0x4b0:0x0] object 0x500000400:8117105 extent [2771862304-2776055807]: client csum d3a08d5c, server csum efbe8302 Mar 16 03:16:55 arkle2 kernel: LustreError: Skipped 17 previous similar messages Mar 16 03:16:55 john7 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448319410432 length 1048576 too big: 1047376 left, 1047376 allowed Mar 16 03:16:55 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e3de8ac00 Mar 16 03:16:55 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e3de8ac00 Mar 16 03:16:55 john17 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594448316861280 length 1048576 too big: 1047576 left, 1047576 allowed Mar 16 03:16:55 john17 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:55 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882bbb34b600 Mar 16 03:16:55 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882bbb34b600 Mar 16 03:16:56 john16 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448321512432 length 1048576 too big: 1048264 left, 1048264 allowed Mar 16 03:16:56 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881e30d81200 Mar 16 03:16:56 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881e30d81200 Mar 16 03:16:56 arkle7 kernel: LustreError: 175513:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff880e9f6b4450 x1594448321512432/t0(0) o4->555ab62f-a8a1-c567-5cc2-8766e0a20f71@192.168.44.116@o2ib44:449/0 lens 608/448 e 0 to 0 dl 1521130659 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:56 arkle7 kernel: LustreError: 175513:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 2 previous similar messages Mar 16 03:16:56 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88189454ec00 Mar 16 03:16:56 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88189454ec00 Mar 16 03:16:56 john21 kernel: LNetError: 895:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448332918640 length 1048576 too big: 1046648 left, 1046648 allowed Mar 16 03:16:56 john21 kernel: LNetError: 895:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:56 arkle7 kernel: Lustre: dagg-OST000d: Bulk IO write error with 555ab62f-a8a1-c567-5cc2-8766e0a20f71 (at 192.168.44.116@o2ib44), client will retry: rc = -110 Mar 16 03:16:56 arkle7 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:16:56 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817696a7200 Mar 16 03:16:56 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817696a7200 Mar 16 03:16:56 arkle3 kernel: LustreError: 187521:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8817d7a7dc50 x1594448332918640/t0(0) o4->2c1132a9-8714-fc22-fa35-2c73e98cd45e@192.168.44.121@o2ib44:449/0 lens 608/448 e 0 to 0 dl 1521130659 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:56 arkle3 kernel: LustreError: 187521:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 1 previous similar message Mar 16 03:16:56 arkle3 kernel: Lustre: dagg-OST0004: Bulk IO write error with 2c1132a9-8714-fc22-fa35-2c73e98cd45e (at 192.168.44.121@o2ib44), client will retry: rc = -110 Mar 16 03:16:56 arkle3 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:16:56 john10 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448319889536 length 1048576 too big: 1046580 left, 1046580 allowed Mar 16 03:16:56 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f46286c00 Mar 16 03:16:56 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f46286c00 Mar 16 03:16:56 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881b22276600 Mar 16 03:16:56 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881b22276600 Mar 16 03:16:57 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f0a188200 Mar 16 03:16:57 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f0a188200 Mar 16 03:16:58 john23 kernel: LNetError: 918:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448282185456 length 1048576 too big: 1046120 left, 1046120 allowed Mar 16 03:16:58 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88210cbaf400 Mar 16 03:16:58 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88210cbaf400 Mar 16 03:16:58 arkle6 kernel: LustreError: 149762:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881d806c7050 x1594448282185456/t0(0) o4->e5e03fa1-aa9e-78b9-d06c-6e18b34089cf@192.168.44.123@o2ib44:451/0 lens 608/448 e 0 to 0 dl 1521130661 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:58 arkle6 kernel: LustreError: 149762:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 2 previous similar messages Mar 16 03:16:58 arkle6 kernel: Lustre: dagg-OST000b: Bulk IO write error with e5e03fa1-aa9e-78b9-d06c-6e18b34089cf (at 192.168.44.123@o2ib44), client will retry: rc = -110 Mar 16 03:16:58 arkle6 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:16:58 arkle6 kernel: LustreError: 168-f: dagg-OST000a: BAD WRITE CHECKSUM: from 12345-192.168.44.123@o2ib44 inode [0x68001c9c3:0x28a:0x0] object 0x600000401:9494173 extent [3193860076-3198050303]: client csum cf15ff3b, server csum 2e5c4a7 Mar 16 03:16:58 arkle6 kernel: LustreError: Skipped 53 previous similar messages Mar 16 03:16:58 arkle7 kernel: LustreError: 168-f: dagg-OST000d: BAD WRITE CHECKSUM: from 12345-192.168.44.118@o2ib44 inode [0x20001066a:0x729:0x0] object 0x0:8477522 extent [3130523800-3134717951]: client csum c544e8b4, server csum 1970d0a Mar 16 03:16:58 arkle7 kernel: LustreError: Skipped 41 previous similar messages Mar 16 03:16:58 john9 kernel: LNetError: 912:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448323976576 length 1048576 too big: 1044656 left, 1044656 allowed Mar 16 03:16:58 john9 kernel: LNetError: 912:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:16:58 arkle4 kernel: LustreError: 168-f: dagg-OST0006: BAD WRITE CHECKSUM: from 12345-192.168.44.114@o2ib44 inode [0x68001c98b:0x47d:0x0] object 0x580000401:9544867 extent [3136153468-3140345855]: client csum 3d3029b6, server csum d440eb87 Mar 16 03:16:58 arkle4 kernel: LustreError: Skipped 17 previous similar messages Mar 16 03:16:58 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880923834000 Mar 16 03:16:58 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880923834000 Mar 16 03:16:58 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ed05ec200 Mar 16 03:16:58 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ed05ec200 Mar 16 03:16:58 john17 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448316863856 length 1048576 too big: 1045604 left, 1045604 allowed Mar 16 03:16:59 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8828c7e39a00 Mar 16 03:16:59 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8828c7e39a00 Mar 16 03:16:59 arkle2 kernel: LustreError: 350108:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88256fc97c50 x1594448316863856/t0(0) o4->6b4209e2-620e-030d-e2d1-b9c575162077@192.168.44.117@o2ib44:451/0 lens 608/448 e 0 to 0 dl 1521130661 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:16:59 arkle2 kernel: LustreError: 350108:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 1 previous similar message Mar 16 03:16:59 john18 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448284396688 length 1048576 too big: 1045960 left, 1045960 allowed Mar 16 03:16:59 arkle2 kernel: Lustre: dagg-OST0003: Bulk IO write error with 6b4209e2-620e-030d-e2d1-b9c575162077 (at 192.168.44.117@o2ib44), client will retry: rc = -110 Mar 16 03:16:59 arkle2 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:16:59 arkle3 kernel: LustreError: 168-f: dagg-OST0005: BAD WRITE CHECKSUM: from 12345-192.168.44.109@o2ib44 inode [0x68001c982:0x487:0x0] object 0x2c0000401:9597154 extent [3436244104-3440435199]: client csum f4980b2d, server csum 2b598048 Mar 16 03:16:59 arkle3 kernel: LustreError: Skipped 31 previous similar messages Mar 16 03:16:59 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8821065a3c00 Mar 16 03:16:59 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8821065a3c00 Mar 16 03:16:59 arkle1 kernel: LustreError: 168-f: dagg-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.44.122@o2ib44 inode [0x28001c1f2:0x264:0x0] object 0x240000400:8165498 extent [2524203472-2528395263]: client csum 6525c304, server csum 5a09cf95 Mar 16 03:16:59 arkle1 kernel: LustreError: Skipped 27 previous similar messages Mar 16 03:16:59 arkle5 kernel: LustreError: 168-f: dagg-OST0009: BAD WRITE CHECKSUM: from 12345-192.168.44.116@o2ib44 inode [0x68001c985:0x44e:0x0] object 0x340000401:9549664 extent [3153702136-3157893119]: client csum 8ae5db6f, server csum 4e1f217b Mar 16 03:16:59 arkle5 kernel: LustreError: Skipped 21 previous similar messages Mar 16 03:17:00 john22 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448280693824 length 1048576 too big: 1047940 left, 1047940 allowed Mar 16 03:17:00 john22 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:00 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881577281a00 Mar 16 03:17:00 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881577281a00 Mar 16 03:17:00 arkle1 kernel: LustreError: 195801:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff880ae0d56450 x1594448280693824/t0(0) o4->53090394-a3ef-6106-82db-77d8ecba9e98@192.168.44.122@o2ib44:452/0 lens 608/448 e 0 to 0 dl 1521130662 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:17:00 arkle1 kernel: LustreError: 195801:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 2 previous similar messages Mar 16 03:17:00 arkle1 kernel: Lustre: dagg-OST0001: Bulk IO write error with 53090394-a3ef-6106-82db-77d8ecba9e98 (at 192.168.44.122@o2ib44), client will retry: rc = -110 Mar 16 03:17:00 arkle1 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:00 john28 kernel: LNetError: 897:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448282279632 length 1048576 too big: 1046944 left, 1046944 allowed Mar 16 03:17:00 john28 kernel: LNetError: 897:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:00 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881fe9676600 Mar 16 03:17:00 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881fe9676600 Mar 16 03:17:00 arkle3 kernel: LustreError: 120480:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8821422c4c50 x1594448282279632/t0(0) o4->0ec03030-04bc-e99c-053e-6e02b04218ff@192.168.44.128@o2ib44:453/0 lens 608/448 e 0 to 0 dl 1521130663 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:17:00 arkle3 kernel: LustreError: 120480:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 1 previous similar message Mar 16 03:17:00 arkle3 kernel: Lustre: dagg-OST0005: Bulk IO write error with 0ec03030-04bc-e99c-053e-6e02b04218ff (at 192.168.44.128@o2ib44), client will retry: rc = -110 Mar 16 03:17:00 arkle3 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:01 john16 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594448321516688 length 1048576 too big: 1046776 left, 1046776 allowed Mar 16 03:17:01 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed85e00 Mar 16 03:17:01 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed85e00 Mar 16 03:17:01 arkle5 kernel: LustreError: 449866:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881619b02450 x1594448321516688/t0(0) o4->555ab62f-a8a1-c567-5cc2-8766e0a20f71@192.168.44.116@o2ib44:454/0 lens 608/448 e 0 to 0 dl 1521130664 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:17:01 arkle5 kernel: LustreError: 449866:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 2 previous similar messages Mar 16 03:17:01 arkle5 kernel: Lustre: dagg-OST0009: Bulk IO write error with 555ab62f-a8a1-c567-5cc2-8766e0a20f71 (at 192.168.44.116@o2ib44), client will retry: rc = -110 Mar 16 03:17:01 arkle5 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:01 john9 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448323979008 length 1048576 too big: 1047780 left, 1047780 allowed Mar 16 03:17:01 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880583af9000 Mar 16 03:17:01 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880583af9000 Mar 16 03:17:03 john19 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448327445664 length 1048576 too big: 1044904 left, 1044904 allowed Mar 16 03:17:03 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881e30d84400 Mar 16 03:17:03 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881e30d84400 Mar 16 03:17:03 arkle7 kernel: LustreError: 175513:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881307760850 x1594448327445664/t0(0) o4->d4acd478-18fd-71d3-c177-089c607f8639@192.168.44.119@o2ib44:456/0 lens 608/448 e 0 to 0 dl 1521130666 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:17:03 arkle7 kernel: LustreError: 175513:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 2 previous similar messages Mar 16 03:17:03 arkle7 kernel: Lustre: dagg-OST000d: Bulk IO write error with d4acd478-18fd-71d3-c177-089c607f8639 (at 192.168.44.119@o2ib44), client will retry: rc = -110 Mar 16 03:17:03 arkle7 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:04 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ae0cec800 Mar 16 03:17:04 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ae0cec800 Mar 16 03:17:04 arkle6 kernel: LustreError: 199667:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8817c201e450 x1594448327446320/t0(0) o4->d4acd478-18fd-71d3-c177-089c607f8639@192.168.44.119@o2ib44:457/0 lens 608/448 e 0 to 0 dl 1521130667 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:17:04 arkle6 kernel: LustreError: 199667:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 2 previous similar messages Mar 16 03:17:04 arkle6 kernel: Lustre: dagg-OST000a: Bulk IO write error with d4acd478-18fd-71d3-c177-089c607f8639 (at 192.168.44.119@o2ib44), client will retry: rc = -110 Mar 16 03:17:04 arkle6 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:07 arkle1 kernel: LustreError: 168-f: dagg-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.44.120@o2ib44 inode [0x28001c1f3:0x22f:0x0] object 0x240000400:8165499 extent [2930006452-2934198271]: client csum eb5e3dfc, server csum 5506137a Mar 16 03:17:07 arkle1 kernel: LustreError: Skipped 26 previous similar messages Mar 16 03:17:11 arkle8 kernel: LustreError: 168-f: dagg-OST000e: BAD WRITE CHECKSUM: from 12345-192.168.44.131@o2ib44 inode [0x68001c9d0:0x20c:0x0] object 0x640000401:9545306 extent [2019034720-2023227391]: client csum 439f54d0, server csum 7084492 Mar 16 03:17:11 arkle8 kernel: LustreError: Skipped 1 previous similar message Mar 16 03:17:11 john26 kernel: LNetError: 904:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448281339792 length 1048576 too big: 1047960 left, 1047960 allowed Mar 16 03:17:11 arkle3 kernel: LustreError: 168-f: dagg-OST0004: BAD WRITE CHECKSUM: from 12345-192.168.44.111@o2ib44 inode [0x200010634:0x4c1:0x0] object 0x0:8489190 extent [3172072732-3176263679]: client csum f052fd4e, server csum 407a90ad Mar 16 03:17:11 arkle3 kernel: LustreError: Skipped 12 previous similar messages Mar 16 03:17:11 arkle6 kernel: LustreError: 168-f: dagg-OST000a: BAD WRITE CHECKSUM: from 12345-192.168.44.111@o2ib44 inode [0x28001c1ba:0x432:0x0] object 0x600000400:8125744 extent [2557826152-2562019327]: client csum 4852f6e7, server csum 2981a05 Mar 16 03:17:11 arkle6 kernel: LustreError: Skipped 20 previous similar messages Mar 16 03:17:11 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880da513d200 Mar 16 03:17:11 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880da513d200 Mar 16 03:17:11 arkle8 kernel: LustreError: 172939:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff880477e8b050 x1594448281339792/t0(0) o4->f4a3cf53-7286-0d27-5cb4-5f6321872642@192.168.44.126@o2ib44:464/0 lens 608/448 e 0 to 0 dl 1521130674 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:17:11 arkle8 kernel: Lustre: dagg-OST000e: Bulk IO write error with f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44), client will retry: rc = -110 Mar 16 03:17:11 john11 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448320168128 length 1048576 too big: 1047284 left, 1047284 allowed Mar 16 03:17:11 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8828303da200 Mar 16 03:17:11 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8828303da200 Mar 16 03:17:11 john25 kernel: LNetError: 904:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448280895840 length 1048576 too big: 1046340 left, 1046340 allowed Mar 16 03:17:11 john25 kernel: LNetError: 904:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:11 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88058bf25200 Mar 16 03:17:11 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88058bf25200 Mar 16 03:17:12 john12 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448324471616 length 1048576 too big: 1046984 left, 1046984 allowed Mar 16 03:17:12 arkle8 kernel: LustreError: 298958:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88058bf23800 Mar 16 03:17:12 arkle8 kernel: LustreError: 298958:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88058bf23800 Mar 16 03:17:12 john31 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594699394190000 length 1048576 too big: 1048208 left, 1048208 allowed Mar 16 03:17:12 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8824cbb58600 Mar 16 03:17:12 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8824cbb58600 Mar 16 03:17:13 john11 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448320175040 length 1048576 too big: 1045348 left, 1045348 allowed Mar 16 03:17:13 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e4405ae00 Mar 16 03:17:13 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e4405ae00 Mar 16 03:17:13 john15 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448323943024 length 1048576 too big: 1046392 left, 1046392 allowed Mar 16 03:17:13 john15 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:13 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8823751f3e00 Mar 16 03:17:13 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8823751f3e00 Mar 16 03:17:14 john11 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448320177872 length 1048576 too big: 1046112 left, 1046112 allowed Mar 16 03:17:14 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881508e12e00 Mar 16 03:17:14 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881508e12e00 Mar 16 03:17:14 arkle3 kernel: LustreError: 148848:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8817d7a78050 x1594448320177872/t0(0) o4->7c2b1b78-121a-b208-fc8a-57295d42d7f7@192.168.44.111@o2ib44:467/0 lens 608/448 e 0 to 0 dl 1521130677 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:17:14 arkle3 kernel: LustreError: 148848:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 1 previous similar message Mar 16 03:17:14 arkle3 kernel: Lustre: dagg-OST0004: Bulk IO write error with 7c2b1b78-121a-b208-fc8a-57295d42d7f7 (at 192.168.44.111@o2ib44), client will retry: rc = -110 Mar 16 03:17:14 arkle3 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:19 arkle8 kernel: LustreError: 168-f: dagg-OST000e: BAD WRITE CHECKSUM: from 12345-192.168.44.111@o2ib44 inode [0x28001c1ba:0x433:0x0] object 0x640000400:8166377 extent [2645987596-2650181631]: client csum 84d37d97, server csum 833b4e04 Mar 16 03:17:19 arkle8 kernel: LustreError: Skipped 68 previous similar messages Mar 16 03:17:23 john11 kernel: LNetError: 904:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448320193440 length 1048576 too big: 1047008 left, 1047008 allowed Mar 16 03:17:23 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88074211d800 Mar 16 03:17:23 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88074211d800 Mar 16 03:17:23 arkle8 kernel: LustreError: 108697:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88052ebb4450 x1594448320193440/t0(0) o4->7c2b1b78-121a-b208-fc8a-57295d42d7f7@192.168.44.111@o2ib44:476/0 lens 608/448 e 0 to 0 dl 1521130686 ref 1 fl Interpret:/0/0 rc 0/0 Mar 16 03:17:23 arkle8 kernel: LustreError: 108697:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 5 previous similar messages Mar 16 03:17:23 arkle8 kernel: Lustre: dagg-OST000e: Bulk IO write error with 7c2b1b78-121a-b208-fc8a-57295d42d7f7 (at 192.168.44.111@o2ib44), client will retry: rc = -110 Mar 16 03:17:23 arkle8 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:17:34 john25 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130610/real 1521130610] req@ffff881574af9e00 x1594448280872016/t0(0) o4->dagg-OST0006-osc-ffff882fd7db5800@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130654 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:34 john25 kernel: Lustre: dagg-OST0006-osc-ffff882fd7db5800: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:34 arkle4 kernel: Lustre: dagg-OST0006: Client 40d9b451-fab3-be40-89fb-0c9b04087671 (at 192.168.44.125@o2ib44) reconnecting Mar 16 03:17:34 arkle4 kernel: Lustre: dagg-OST0006: Connection restored to edb2edda-7927-928c-4497-0706e1cfa6ce (at 192.168.44.125@o2ib44) Mar 16 03:17:34 john25 kernel: Lustre: dagg-OST0006-osc-ffff882fd7db5800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:17:34 john25 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448280900944 length 1048576 too big: 1048092 left, 1048092 allowed Mar 16 03:17:34 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882065dd0800 Mar 16 03:17:34 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882065dd0800 Mar 16 03:17:34 arkle4 kernel: LustreError: 302876:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882ddb73f050 x1594448280872016/t0(0) o4->40d9b451-fab3-be40-89fb-0c9b04087671@192.168.44.125@o2ib44:487/0 lens 608/448 e 0 to 0 dl 1521130697 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:17:34 john30 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130610/real 1521130610] req@ffff882e4338a400 x1594699400481984/t0(0) o4->dagg-OST000c-osc-ffff88189ac92000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130654 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:34 john30 kernel: Lustre: dagg-OST000c-osc-ffff88189ac92000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:34 arkle7 kernel: Lustre: dagg-OST000c: Client faed83b4-3a5e-2a8e-2882-7e99d174dfb0 (at 192.168.44.130@o2ib44) reconnecting Mar 16 03:17:34 arkle7 kernel: Lustre: dagg-OST000c: Connection restored to 2beb6461-856d-b027-2da8-040c1c990221 (at 192.168.44.130@o2ib44) Mar 16 03:17:34 arkle4 kernel: LustreError: 302876:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 4 previous similar messages Mar 16 03:17:34 arkle4 kernel: Lustre: dagg-OST0006: Bulk IO write error with 40d9b451-fab3-be40-89fb-0c9b04087671 (at 192.168.44.125@o2ib44), client will retry: rc = -110 Mar 16 03:17:34 arkle4 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:17:34 john30 kernel: Lustre: dagg-OST000c-osc-ffff88189ac92000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:34 john30 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594699400511168 length 1048576 too big: 1048516 left, 1048516 allowed Mar 16 03:17:34 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881786098200 Mar 16 03:17:34 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881786098200 Mar 16 03:17:34 arkle7 kernel: LustreError: 34887:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882f105be050 x1594699400481984/t0(0) o4->faed83b4-3a5e-2a8e-2882-7e99d174dfb0@192.168.44.130@o2ib44:487/0 lens 608/448 e 0 to 0 dl 1521130697 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:17:34 john27 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130610/real 1521130610] req@ffff881b4e2d3600 x1594448285369712/t0(0) o4->dagg-OST000f-osc-ffff88015a1e4800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130654 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:34 john27 kernel: Lustre: dagg-OST000f-osc-ffff88015a1e4800: Connection to dagg-OST000f (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:34 john28 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130610/real 1521130610] req@ffff881aaab92d00 x1594448282246752/t0(0) o4->dagg-OST0006-osc-ffff8817d9cf4000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130654 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:34 john28 kernel: Lustre: dagg-OST0006-osc-ffff8817d9cf4000: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:34 arkle8 kernel: Lustre: dagg-OST000f: Client 219d9f55-42f0-6656-ab6f-f4de9ef8c6c7 (at 192.168.44.127@o2ib44) reconnecting Mar 16 03:17:34 arkle8 kernel: Lustre: dagg-OST000f: Connection restored to c8b7715d-04bb-a76a-3db9-c45127314e23 (at 192.168.44.127@o2ib44) Mar 16 03:17:34 arkle7 kernel: Lustre: dagg-OST000c: Bulk IO write error with faed83b4-3a5e-2a8e-2882-7e99d174dfb0 (at 192.168.44.130@o2ib44), client will retry: rc = -110 Mar 16 03:17:34 john28 kernel: Lustre: dagg-OST0006-osc-ffff8817d9cf4000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:17:34 john28 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448282303456 length 1048576 too big: 1045344 left, 1045344 allowed Mar 16 03:17:34 john27 kernel: Lustre: dagg-OST000f-osc-ffff88015a1e4800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:17:34 john27 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448285415120 length 1048576 too big: 1046096 left, 1046096 allowed Mar 16 03:17:34 john27 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:17:34 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882065dd0800 Mar 16 03:17:34 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882065dd0800 Mar 16 03:17:34 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ec2255c00 Mar 16 03:17:34 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ec2255c00 Mar 16 03:17:34 arkle8 kernel: LustreError: 303900:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8805a52ef050 x1594448285369712/t0(0) o4->219d9f55-42f0-6656-ab6f-f4de9ef8c6c7@192.168.44.127@o2ib44:487/0 lens 608/448 e 0 to 0 dl 1521130697 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:17:34 john20 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130610/real 1521130610] req@ffff8828a2b52100 x1594448327525968/t0(0) o4->dagg-OST000d-osc-ffff8817da6ad800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130654 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:34 john20 kernel: Lustre: dagg-OST000d-osc-ffff8817da6ad800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:34 arkle8 kernel: Lustre: dagg-OST000f: Bulk IO write error with 219d9f55-42f0-6656-ab6f-f4de9ef8c6c7 (at 192.168.44.127@o2ib44), client will retry: rc = -110 Mar 16 03:17:34 john20 kernel: Lustre: dagg-OST000d-osc-ffff8817da6ad800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:34 john20 kernel: LNetError: 897:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448327601392 length 1048576 too big: 1045364 left, 1045364 allowed Mar 16 03:17:34 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881786098200 Mar 16 03:17:34 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881786098200 Mar 16 03:17:34 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88179472ba00 Mar 16 03:17:34 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8e400 Mar 16 03:17:34 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88179472ba00 Mar 16 03:17:34 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8e400 Mar 16 03:17:34 john21 kernel: Lustre: 918:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130610/real 1521130610] req@ffff8813c2e68300 x1594448332914000/t0(0) o4->dagg-OST000c-osc-ffff882fc5588800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130654 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:34 john21 kernel: Lustre: dagg-OST000c-osc-ffff882fc5588800: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:34 john21 kernel: Lustre: dagg-OST000c-osc-ffff882fc5588800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:34 john21 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448332942592 length 1048576 too big: 1044712 left, 1044712 allowed Mar 16 03:17:34 john14 kernel: Lustre: 949:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130610/real 1521130610] req@ffff882aca12b900 x1594448317081440/t0(0) o4->dagg-OST000c-osc-ffff88189a368800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130654 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:34 john14 kernel: Lustre: dagg-OST0006-osc-ffff88189a368800: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:34 john14 kernel: Lustre: dagg-OST0006-osc-ffff88189a368800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:17:34 john14 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448317127488 length 1048576 too big: 1047896 left, 1047896 allowed Mar 16 03:17:34 john14 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:34 john14 kernel: Lustre: 949:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:34 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88091d4afe00 Mar 16 03:17:34 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88091d4afe00 Mar 16 03:17:34 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88091d4afe00 Mar 16 03:17:34 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88091d4afe00 Mar 16 03:17:34 john15 kernel: Lustre: 949:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130610/real 1521130610] req@ffff88207ef11200 x1594448323936064/t0(0) o4->dagg-OST000d-osc-ffff882fde9f4800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130654 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:34 john15 kernel: Lustre: dagg-OST000d-osc-ffff882fde9f4800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:34 john15 kernel: Lustre: dagg-OST000d-osc-ffff882fde9f4800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:34 john15 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448323965056 length 1048576 too big: 1045944 left, 1045944 allowed Mar 16 03:17:34 john24 kernel: Lustre: 951:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130610/real 1521130610] req@ffff882c67f3f500 x1594448284013248/t0(0) o4->dagg-OST000c-osc-ffff882fcb645000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130654 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:34 john24 kernel: Lustre: dagg-OST000c-osc-ffff882fcb645000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:34 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882900149200 Mar 16 03:17:34 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882900149200 Mar 16 03:17:34 john24 kernel: Lustre: dagg-OST000c-osc-ffff882fcb645000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:34 john22 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130610/real 1521130610] req@ffff8823b403b900 x1594448280685024/t0(0) o4->dagg-OST000c-osc-ffff88189a897000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130654 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:34 john24 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448284059872 length 1048576 too big: 1046748 left, 1046748 allowed Mar 16 03:17:34 john24 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 2 previous similar messages Mar 16 03:17:34 john22 kernel: Lustre: dagg-OST000c-osc-ffff88189a897000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:34 john22 kernel: Lustre: dagg-OST000c-osc-ffff88189a897000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:34 john7 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130610/real 1521130610] req@ffff8817a06ff500 x1594448319400640/t0(0) o4->dagg-OST0006-osc-ffff882fda4b4000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130654 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:34 john7 kernel: Lustre: dagg-OST0006-osc-ffff882fda4b4000: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:34 john22 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448280717040 length 1048576 too big: 1048436 left, 1048436 allowed Mar 16 03:17:34 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88091d4afe00 Mar 16 03:17:34 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88091d4afe00 Mar 16 03:17:34 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c51845e00 Mar 16 03:17:34 arkle4 kernel: Lustre: dagg-OST0006: Client 95b9add5-8117-706c-dad4-c3e4c830ffa8 (at 192.168.44.107@o2ib44) reconnecting Mar 16 03:17:34 arkle4 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:34 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c51845e00 Mar 16 03:17:34 arkle4 kernel: Lustre: dagg-OST0006: Connection restored to 0ec19cf2-d52d-2768-a091-35636b4aa0f5 (at 192.168.44.107@o2ib44) Mar 16 03:17:34 arkle4 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:34 john30 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130610/real 1521130610] req@ffff882e43388600 x1594699400484240/t0(0) o4->dagg-OST0006-osc-ffff88189ac92000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130654 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:34 john30 kernel: Lustre: dagg-OST0006-osc-ffff88189ac92000: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:34 john30 kernel: Lustre: dagg-OST0006-osc-ffff88189ac92000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:17:34 john7 kernel: Lustre: dagg-OST0006-osc-ffff882fda4b4000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:17:34 john7 kernel: LNetError: 914:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448319435616 length 1048576 too big: 1045464 left, 1045464 allowed Mar 16 03:17:34 john7 kernel: LNetError: 914:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:35 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882efa9aec00 Mar 16 03:17:35 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882efa9aec00 Mar 16 03:17:35 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881751ca3e00 Mar 16 03:17:35 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ec2255c00 Mar 16 03:17:35 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ec2255c00 Mar 16 03:17:35 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881751ca3e00 Mar 16 03:17:35 john6 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130612] req@ffff8803cff98300 x1594448323930016/t0(0) o4->dagg-OST0002-osc-ffff881899c73800@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john6 kernel: Lustre: dagg-OST0002-osc-ffff881899c73800: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 john26 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff882a53e52700 x1594448281318416/t0(0) o4->dagg-OST0008-osc-ffff882fde832000@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john26 kernel: Lustre: dagg-OST0008-osc-ffff882fde832000: Connection to dagg-OST0008 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 arkle2 kernel: Lustre: dagg-OST0002: Client efbad07f-625e-4fc5-c94b-ef8442fed4fb (at 192.168.44.106@o2ib44) reconnecting Mar 16 03:17:35 arkle2 kernel: Lustre: dagg-OST0002: Connection restored to 3c1483cd-dc70-d4a2-d9df-82049cf8a62d (at 192.168.44.106@o2ib44) Mar 16 03:17:35 arkle5 kernel: Lustre: dagg-OST0008: Client f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44) reconnecting Mar 16 03:17:35 arkle5 kernel: Lustre: dagg-OST0008: Connection restored to d5de0e2d-4431-41fa-738a-2fa7d0a3c365 (at 192.168.44.126@o2ib44) Mar 16 03:17:35 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881dd10d1600 Mar 16 03:17:35 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881dd10d1600 Mar 16 03:17:35 arkle2 kernel: LustreError: 349954:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88151bb29450 x1594448323930016/t0(0) o4->efbad07f-625e-4fc5-c94b-ef8442fed4fb@192.168.44.106@o2ib44:488/0 lens 608/448 e 0 to 0 dl 1521130698 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:17:35 john26 kernel: Lustre: dagg-OST0008-osc-ffff882fde832000: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:17:35 john18 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff882b86330900 x1594448284367264/t0(0) o4->dagg-OST0000-osc-ffff88015a243000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john18 kernel: Lustre: dagg-OST0000-osc-ffff88015a243000: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 john20 kernel: Lustre: 918:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff8803a6e11b00 x1594448327528752/t0(0) o4->dagg-OST0005-osc-ffff8817da6ad800@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john20 kernel: Lustre: dagg-OST0005-osc-ffff8817da6ad800: Connection to dagg-OST0005 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 john26 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594448281362704 length 1048576 too big: 1045572 left, 1045572 allowed Mar 16 03:17:35 john6 kernel: Lustre: dagg-OST0002-osc-ffff881899c73800: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:17:35 john6 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448323976880 length 1048576 too big: 1044832 left, 1044832 allowed Mar 16 03:17:35 john6 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:35 arkle1 kernel: Lustre: dagg-OST0000: Client 5abb6bc1-b79c-bc3d-60a6-8103bee751c7 (at 192.168.44.118@o2ib44) reconnecting Mar 16 03:17:35 arkle1 kernel: Lustre: dagg-OST0000: Connection restored to aad63549-8ffd-bbaa-adc4-c670c37277c9 (at 192.168.44.118@o2ib44) Mar 16 03:17:35 arkle3 kernel: Lustre: dagg-OST0005: Client a5139128-ee9e-5a55-5ed0-cfdb3f00b0b5 (at 192.168.44.120@o2ib44) reconnecting Mar 16 03:17:35 arkle3 kernel: Lustre: dagg-OST0005: Connection restored to 8dcc63fb-de49-8cb9-bf59-b2315da365d4 (at 192.168.44.120@o2ib44) Mar 16 03:17:35 arkle2 kernel: Lustre: dagg-OST0002: Bulk IO write error with efbad07f-625e-4fc5-c94b-ef8442fed4fb (at 192.168.44.106@o2ib44), client will retry: rc = -110 Mar 16 03:17:35 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed81600 Mar 16 03:17:35 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed81600 Mar 16 03:17:35 arkle5 kernel: LustreError: 449866:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88141e1dac50 x1594448281318416/t0(0) o4->f4a3cf53-7286-0d27-5cb4-5f6321872642@192.168.44.126@o2ib44:488/0 lens 608/448 e 0 to 0 dl 1521130698 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:17:35 john20 kernel: Lustre: dagg-OST0005-osc-ffff8817da6ad800: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:17:35 john18 kernel: Lustre: dagg-OST0000-osc-ffff88015a243000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:17:35 john18 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448284419856 length 1048576 too big: 1048556 left, 1048556 allowed Mar 16 03:17:35 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c282f9c00 Mar 16 03:17:35 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c282f9c00 Mar 16 03:17:35 arkle3 kernel: LustreError: 120216:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882a77d28050 x1594448327528752/t0(0) o4->a5139128-ee9e-5a55-5ed0-cfdb3f00b0b5@192.168.44.120@o2ib44:488/0 lens 608/448 e 0 to 0 dl 1521130698 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:17:35 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac955200 Mar 16 03:17:35 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac955200 Mar 16 03:17:35 arkle1 kernel: LustreError: 395412:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882fc14be850 x1594448284367264/t0(0) o4->5abb6bc1-b79c-bc3d-60a6-8103bee751c7@192.168.44.118@o2ib44:488/0 lens 608/448 e 0 to 0 dl 1521130698 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:17:35 john23 kernel: Lustre: 924:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff8814da225d00 x1594448282171968/t0(0) o4->dagg-OST0002-osc-ffff882fdbc22800@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john23 kernel: Lustre: dagg-OST0002-osc-ffff882fdbc22800: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881751ca3e00 Mar 16 03:17:35 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881751ca3e00 Mar 16 03:17:35 arkle5 kernel: Lustre: dagg-OST0008: Bulk IO write error with f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44), client will retry: rc = -110 Mar 16 03:17:35 john23 kernel: Lustre: dagg-OST0002-osc-ffff882fdbc22800: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:17:35 arkle3 kernel: Lustre: dagg-OST0005: Bulk IO write error with a5139128-ee9e-5a55-5ed0-cfdb3f00b0b5 (at 192.168.44.120@o2ib44), client will retry: rc = -110 Mar 16 03:17:35 john23 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448282209504 length 1048576 too big: 1047668 left, 1047668 allowed Mar 16 03:17:35 john23 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:35 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e93e3b200 Mar 16 03:17:35 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e93e3b200 Mar 16 03:17:35 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cd200 Mar 16 03:17:35 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cd200 Mar 16 03:17:35 arkle1 kernel: Lustre: dagg-OST0000: Bulk IO write error with efbad07f-625e-4fc5-c94b-ef8442fed4fb (at 192.168.44.106@o2ib44), client will retry: rc = -110 Mar 16 03:17:35 arkle1 kernel: LustreError: 395412:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 1 previous similar message Mar 16 03:17:35 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e097200 Mar 16 03:17:35 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e097200 Mar 16 03:17:35 john8 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff88270812b300 x1594448316803760/t0(0) o4->dagg-OST0000-osc-ffff88015a2fe000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john8 kernel: Lustre: dagg-OST0000-osc-ffff88015a2fe000: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 john16 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff881e67300f00 x1594448321487984/t0(0) o4->dagg-OST0007-osc-ffff882fc87e8000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john16 kernel: Lustre: dagg-OST0007-osc-ffff882fc87e8000: Connection to dagg-OST0007 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 john16 kernel: Lustre: dagg-OST0007-osc-ffff882fc87e8000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:17:35 john16 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448321540016 length 1048576 too big: 1046104 left, 1046104 allowed Mar 16 03:17:35 john16 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:35 john25 kernel: Lustre: 929:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff88038df1aa00 x1594448280874624/t0(0) o4->dagg-OST000f-osc-ffff882fd7db5800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john25 kernel: Lustre: dagg-OST000f-osc-ffff882fd7db5800: Connection to dagg-OST000f (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e093a00 Mar 16 03:17:35 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e093a00 Mar 16 03:17:35 arkle8 kernel: Lustre: dagg-OST000f: Connection restored to (at 192.168.44.125@o2ib44) Mar 16 03:17:35 arkle8 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:35 john25 kernel: Lustre: dagg-OST000f-osc-ffff882fd7db5800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:17:35 john8 kernel: Lustre: dagg-OST0000-osc-ffff88015a2fe000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:17:35 john8 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448316833920 length 1048576 too big: 1047284 left, 1047284 allowed Mar 16 03:17:35 john8 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:35 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e1b697000 Mar 16 03:17:35 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e1b697000 Mar 16 03:17:35 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cee00 Mar 16 03:17:35 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cee00 Mar 16 03:17:35 john24 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff881381758f00 x1594448284017648/t0(0) o4->dagg-OST0000-osc-ffff882fcb645000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john24 kernel: Lustre: dagg-OST0000-osc-ffff882fcb645000: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 john24 kernel: Lustre: dagg-OST0000-osc-ffff882fcb645000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:17:35 john29 kernel: Lustre: 922:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff8801acec4800 x1594699419068752/t0(0) o4->dagg-OST0002-osc-ffff88189afee000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john33 kernel: Lustre: 940:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff882e3f367500 x1594699393878992/t0(0) o4->dagg-OST000a-osc-ffff88006bb84800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john33 kernel: Lustre: dagg-OST000a-osc-ffff88006bb84800: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882474794600 Mar 16 03:17:35 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882474794600 Mar 16 03:17:35 john29 kernel: Lustre: dagg-OST0002-osc-ffff88189afee000: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 john29 kernel: Lustre: dagg-OST0002-osc-ffff88189afee000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:17:35 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cee00 Mar 16 03:17:35 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cee00 Mar 16 03:17:35 john29 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594699419093888 length 1048576 too big: 1048152 left, 1048152 allowed Mar 16 03:17:35 arkle6 kernel: Lustre: dagg-OST000a: Client 22c507bc-bb1f-dbcf-8316-529c1039bbfe (at 192.168.44.133@o2ib44) reconnecting Mar 16 03:17:35 arkle6 kernel: Lustre: dagg-OST000a: Connection restored to 6e809e98-956d-fb0a-6999-afce652c717c (at 192.168.44.133@o2ib44) Mar 16 03:17:35 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881dd10d1600 Mar 16 03:17:35 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881dd10d1600 Mar 16 03:17:35 john32 kernel: Lustre: 928:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff882182fbef00 x1594699396464000/t0(0) o4->dagg-OST000d-osc-ffff8817c5384000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john32 kernel: Lustre: dagg-OST000d-osc-ffff8817c5384000: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 arkle7 kernel: Lustre: dagg-OST000d: Client 0202600d-b066-c547-a639-ee53deb97148 (at 192.168.44.132@o2ib44) reconnecting Mar 16 03:17:35 arkle7 kernel: Lustre: Skipped 6 previous similar messages Mar 16 03:17:35 arkle7 kernel: Lustre: dagg-OST000d: Connection restored to ac18b8dd-5f9b-d1d9-555f-48ee1cd5b8ea (at 192.168.44.132@o2ib44) Mar 16 03:17:35 arkle7 kernel: Lustre: Skipped 6 previous similar messages Mar 16 03:17:35 john33 kernel: Lustre: dagg-OST000a-osc-ffff88006bb84800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:17:35 john33 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594699393909712 length 1048576 too big: 1046348 left, 1046348 allowed Mar 16 03:17:35 john33 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:35 john9 kernel: Lustre: 935:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff88258cc91b00 x1594448323945264/t0(0) o4->dagg-OST0004-osc-ffff882fdd33b800@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john32 kernel: Lustre: dagg-OST000d-osc-ffff8817c5384000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:35 john12 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff8809853abc00 x1594448324464560/t0(0) o4->dagg-OST000a-osc-ffff88015a3ab800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john12 kernel: Lustre: dagg-OST000a-osc-ffff88015a3ab800: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8821a45ac800 Mar 16 03:17:35 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8821a45ac800 Mar 16 03:17:35 arkle6 kernel: LustreError: 386594:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882c3d1bf850 x1594699393878992/t0(0) o4->22c507bc-bb1f-dbcf-8316-529c1039bbfe@192.168.44.133@o2ib44:488/0 lens 608/448 e 0 to 0 dl 1521130698 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:17:35 john9 kernel: Lustre: dagg-OST0004-osc-ffff882fdd33b800: Connection to dagg-OST0004 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 john9 kernel: Lustre: dagg-OST0004-osc-ffff882fdd33b800: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:17:35 john12 kernel: Lustre: dagg-OST000a-osc-ffff88015a3ab800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:17:35 john27 kernel: Lustre: 920:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff880731269800 x1594448285372496/t0(0) o4->dagg-OST0000-osc-ffff88015a1e4800@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john32 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594699396492128 length 1048576 too big: 1048540 left, 1048540 allowed Mar 16 03:17:35 john32 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:35 john9 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448324002512 length 1048576 too big: 1044696 left, 1044696 allowed Mar 16 03:17:35 john27 kernel: Lustre: dagg-OST0000-osc-ffff88015a1e4800: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 john27 kernel: Lustre: dagg-OST0000-osc-ffff88015a1e4800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:17:35 arkle6 kernel: LustreError: 386594:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 1 previous similar message Mar 16 03:17:35 arkle6 kernel: Lustre: dagg-OST000a: Bulk IO write error with 22c507bc-bb1f-dbcf-8316-529c1039bbfe (at 192.168.44.133@o2ib44), client will retry: rc = -110 Mar 16 03:17:35 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881e30d81a00 Mar 16 03:17:35 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881e30d81a00 Mar 16 03:17:35 john15 kernel: Lustre: 941:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff882274ba8600 x1594448323938480/t0(0) o4->dagg-OST0008-osc-ffff882fde9f4800@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john15 kernel: Lustre: dagg-OST0008-osc-ffff882fde9f4800: Connection to dagg-OST0008 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894aa800 Mar 16 03:17:35 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894aa800 Mar 16 03:17:35 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cee00 Mar 16 03:17:35 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cee00 Mar 16 03:17:35 arkle6 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:35 john12 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448324493824 length 1048576 too big: 1047560 left, 1047560 allowed Mar 16 03:17:35 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ed05ede00 Mar 16 03:17:35 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ed05ede00 Mar 16 03:17:35 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817583a1a00 Mar 16 03:17:35 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808913f9200 Mar 16 03:17:35 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c65f1da00 Mar 16 03:17:35 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c65f1da00 Mar 16 03:17:35 arkle5 kernel: Lustre: dagg-OST0008: Client 5cb90e1d-9f04-23d7-6ef7-f040ab7da8f3 (at 192.168.44.115@o2ib44) reconnecting Mar 16 03:17:35 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817583a1a00 Mar 16 03:17:35 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808913f9200 Mar 16 03:17:35 arkle5 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:35 arkle5 kernel: Lustre: dagg-OST0008: Connection restored to e3d848c2-a44b-2da7-d95d-7c7b7657b642 (at 192.168.44.115@o2ib44) Mar 16 03:17:35 john13 kernel: Lustre: 935:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff88283c641800 x1594448319199040/t0(0) o4->dagg-OST000d-osc-ffff88015a2c6800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john13 kernel: Lustre: dagg-OST000d-osc-ffff88015a2c6800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 arkle5 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:35 john13 kernel: Lustre: dagg-OST000d-osc-ffff88015a2c6800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:35 john15 kernel: Lustre: dagg-OST0008-osc-ffff882fde9f4800: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:17:35 john13 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448319242688 length 1048576 too big: 1046108 left, 1046108 allowed Mar 16 03:17:35 john21 kernel: Lustre: 928:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff8827cc377800 x1594448332914704/t0(0) o4->dagg-OST0004-osc-ffff882fc5588800@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john21 kernel: Lustre: 928:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:35 john21 kernel: Lustre: dagg-OST0004-osc-ffff882fc5588800: Connection to dagg-OST0004 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c65f1da00 Mar 16 03:17:35 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c65f1da00 Mar 16 03:17:35 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881769c82400 Mar 16 03:17:35 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881769c82400 Mar 16 03:17:35 john21 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:35 john22 kernel: Lustre: 938:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff8818ffaad700 x1594448280687712/t0(0) o4->dagg-OST000b-osc-ffff88189a897000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john22 kernel: Lustre: dagg-OST000b-osc-ffff88189a897000: Connection to dagg-OST000b (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 john22 kernel: Lustre: dagg-OST000b-osc-ffff88189a897000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:17:35 arkle3 kernel: Lustre: dagg-OST0004: Client 2c1132a9-8714-fc22-fa35-2c73e98cd45e (at 192.168.44.121@o2ib44) reconnecting Mar 16 03:17:35 arkle3 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:17:35 arkle3 kernel: Lustre: dagg-OST0004: Connection restored to 8150df4b-630f-9694-b32f-69d5f5733e40 (at 192.168.44.121@o2ib44) Mar 16 03:17:35 arkle3 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:17:35 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ae0ceca00 Mar 16 03:17:35 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ae0ceca00 Mar 16 03:17:35 john21 kernel: Lustre: dagg-OST0004-osc-ffff882fc5588800: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:17:35 john21 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:35 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817583a1a00 Mar 16 03:17:35 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817583a1a00 Mar 16 03:17:35 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808913f9200 Mar 16 03:17:35 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808913f9200 Mar 16 03:17:35 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894ac600 Mar 16 03:17:35 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894ac600 Mar 16 03:17:35 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e093a00 Mar 16 03:17:35 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e093a00 Mar 16 03:17:35 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808913f9200 Mar 16 03:17:35 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808913f9200 Mar 16 03:17:35 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e093a00 Mar 16 03:17:35 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e093a00 Mar 16 03:17:35 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ae0ceca00 Mar 16 03:17:35 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ae0ceca00 Mar 16 03:17:35 john19 kernel: Lustre: 941:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff880808246c00 x1594448327426912/t0(0) o4->dagg-OST000a-osc-ffff88189a323000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john19 kernel: Lustre: dagg-OST000a-osc-ffff88189a323000: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 john19 kernel: Lustre: dagg-OST000a-osc-ffff88189a323000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:17:35 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ae0ceca00 Mar 16 03:17:35 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177695f600 Mar 16 03:17:35 john17 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff880e00b31500 x1594448316853392/t0(0) o4->dagg-OST0002-osc-ffff881899e5b000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:35 john17 kernel: Lustre: dagg-OST0002-osc-ffff881899e5b000: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:35 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ae0ceca00 Mar 16 03:17:35 arkle2 kernel: Lustre: dagg-OST0002: Client 6b4209e2-620e-030d-e2d1-b9c575162077 (at 192.168.44.117@o2ib44) reconnecting Mar 16 03:17:35 arkle2 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:35 arkle2 kernel: Lustre: dagg-OST0002: Connection restored to 7002ccab-a112-d1d5-6fdf-c8c42ad342fc (at 192.168.44.117@o2ib44) Mar 16 03:17:35 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177695f600 Mar 16 03:17:35 john19 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448327469248 length 1048576 too big: 1047928 left, 1047928 allowed Mar 16 03:17:35 john19 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:35 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed81600 Mar 16 03:17:35 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed81600 Mar 16 03:17:35 arkle2 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:35 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88202fc8f000 Mar 16 03:17:35 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88202fc8f000 Mar 16 03:17:35 john17 kernel: Lustre: dagg-OST0002-osc-ffff881899e5b000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:17:35 john17 kernel: LNetError: 912:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448316904352 length 1048576 too big: 1044508 left, 1044508 allowed Mar 16 03:17:36 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed81600 Mar 16 03:17:36 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed81600 Mar 16 03:17:36 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e93e3b200 Mar 16 03:17:36 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e93e3b200 Mar 16 03:17:36 john31 kernel: Lustre: 910:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff88158028aa00 x1594699394166672/t0(0) o4->dagg-OST0008-osc-ffff882fde9fd000@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john31 kernel: Lustre: dagg-OST0008-osc-ffff882fde9fd000: Connection to dagg-OST0008 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 john31 kernel: Lustre: dagg-OST0008-osc-ffff882fde9fd000: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:17:36 john31 kernel: LNetError: 895:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594699394199296 length 1048576 too big: 1048064 left, 1048064 allowed Mar 16 03:17:36 john18 kernel: Lustre: 921:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff8804aef68600 x1594448284370160/t0(0) o4->dagg-OST0008-osc-ffff88015a243000@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john18 kernel: Lustre: dagg-OST0008-osc-ffff88015a243000: Connection to dagg-OST0008 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c65f1da00 Mar 16 03:17:36 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c65f1da00 Mar 16 03:17:36 john18 kernel: Lustre: dagg-OST0008-osc-ffff88015a243000: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:17:36 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c65f1da00 Mar 16 03:17:36 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c65f1da00 Mar 16 03:17:36 john23 kernel: Lustre: 935:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff88160dfb8600 x1594448282176160/t0(0) o4->dagg-OST0001-osc-ffff882fdbc22800@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john23 kernel: Lustre: dagg-OST0001-osc-ffff882fdbc22800: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 john26 kernel: Lustre: 915:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130611/real 1521130611] req@ffff881769d13300 x1594448281315440/t0(0) o4->dagg-OST0009-osc-ffff882fde832000@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john26 kernel: Lustre: dagg-OST0009-osc-ffff882fde832000: Connection to dagg-OST0009 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 john26 kernel: Lustre: dagg-OST0009-osc-ffff882fde832000: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:17:36 arkle1 kernel: Lustre: dagg-OST0001: Client e5e03fa1-aa9e-78b9-d06c-6e18b34089cf (at 192.168.44.123@o2ib44) reconnecting Mar 16 03:17:36 arkle1 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:17:36 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881dd10d1600 Mar 16 03:17:36 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ecf6fc800 Mar 16 03:17:36 john9 kernel: Lustre: 949:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff8822bf6a2a00 x1594448323950288/t0(0) o4->dagg-OST0005-osc-ffff882fdd33b800@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john9 kernel: Lustre: dagg-OST0005-osc-ffff882fdd33b800: Connection to dagg-OST0005 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 john9 kernel: Lustre: dagg-OST0005-osc-ffff882fdd33b800: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:17:36 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881dd10d1600 Mar 16 03:17:36 arkle1 kernel: Lustre: dagg-OST0001: Connection restored to 1fea7452-ea20-7788-5be4-f826c8c7f864 (at 192.168.44.123@o2ib44) Mar 16 03:17:36 arkle1 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:17:36 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894a8800 Mar 16 03:17:36 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894a8800 Mar 16 03:17:36 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed81600 Mar 16 03:17:36 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed81600 Mar 16 03:17:36 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ecf6fc800 Mar 16 03:17:36 john23 kernel: Lustre: dagg-OST0001-osc-ffff882fdbc22800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:17:36 john33 kernel: Lustre: 942:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff881d9eb42d00 x1594699393883264/t0(0) o4->dagg-OST0003-osc-ffff88006bb84800@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john33 kernel: Lustre: dagg-OST0003-osc-ffff88006bb84800: Connection to dagg-OST0003 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dc52f1a00 Mar 16 03:17:36 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dc52f1a00 Mar 16 03:17:36 john33 kernel: Lustre: dagg-OST0003-osc-ffff88006bb84800: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:17:36 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac955200 Mar 16 03:17:36 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac955200 Mar 16 03:17:36 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ed2786000 Mar 16 03:17:36 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ed2786000 Mar 16 03:17:36 john8 kernel: Lustre: 923:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff88150db1cb00 x1594448316806480/t0(0) o4->dagg-OST0003-osc-ffff88015a2fe000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john8 kernel: Lustre: dagg-OST0003-osc-ffff88015a2fe000: Connection to dagg-OST0003 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 john8 kernel: Lustre: dagg-OST0003-osc-ffff88015a2fe000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:17:36 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ecf6fc800 Mar 16 03:17:36 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ecf6fc800 Mar 16 03:17:36 john24 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff88138175d700 x1594448284018560/t0(0) o4->dagg-OST0001-osc-ffff882fcb645000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john24 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:36 john24 kernel: Lustre: dagg-OST0001-osc-ffff882fcb645000: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 john24 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:36 john24 kernel: Lustre: dagg-OST0001-osc-ffff882fcb645000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:17:36 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881dd10d1600 Mar 16 03:17:36 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ed2781400 Mar 16 03:17:36 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ed2781400 Mar 16 03:17:36 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881dd10d1600 Mar 16 03:17:36 john24 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:36 john25 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff881a57c36000 x1594448280876144/t0(0) o4->dagg-OST0002-osc-ffff882fd7db5800@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john25 kernel: Lustre: dagg-OST0002-osc-ffff882fd7db5800: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 john25 kernel: Lustre: dagg-OST0002-osc-ffff882fd7db5800: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:17:36 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cb200 Mar 16 03:17:36 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cb200 Mar 16 03:17:36 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9767400 Mar 16 03:17:36 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9767400 Mar 16 03:17:36 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cb200 Mar 16 03:17:36 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cb200 Mar 16 03:17:36 john20 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff882eb0eef800 x1594448327532816/t0(0) o4->dagg-OST0001-osc-ffff8817da6ad800@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john20 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:36 arkle6 kernel: Lustre: dagg-OST000b: Client 7d6768d1-f792-7c54-7167-7223c4aae84c (at 192.168.44.109@o2ib44) reconnecting Mar 16 03:17:36 john20 kernel: Lustre: dagg-OST0001-osc-ffff8817da6ad800: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 john20 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:36 john20 kernel: Lustre: dagg-OST0001-osc-ffff8817da6ad800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:17:36 john7 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff881951f74500 x1594448319405744/t0(0) o4->dagg-OST0001-osc-ffff882fda4b4000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john7 kernel: Lustre: dagg-OST0001-osc-ffff882fda4b4000: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 arkle6 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:17:36 arkle6 kernel: Lustre: dagg-OST000b: Connection restored to f8702301-42cb-b7cb-3022-85cf78a84aab (at 192.168.44.109@o2ib44) Mar 16 03:17:36 arkle6 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:17:36 john20 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:36 john7 kernel: Lustre: dagg-OST0001-osc-ffff882fda4b4000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:17:36 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ed05eec00 Mar 16 03:17:36 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ed05eec00 Mar 16 03:17:36 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817deeb8000 Mar 16 03:17:36 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817deeb8000 Mar 16 03:17:36 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88263d2f2600 Mar 16 03:17:36 john29 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff88195e6b8c00 x1594699419069488/t0(0) o4->dagg-OST000d-osc-ffff88189afee000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john29 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:36 john29 kernel: Lustre: dagg-OST000d-osc-ffff88189afee000: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 john28 kernel: Lustre: 911:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff880a09ad8600 x1594448282252368/t0(0) o4->dagg-OST0002-osc-ffff8817d9cf4000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john28 kernel: Lustre: dagg-OST0002-osc-ffff8817d9cf4000: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88263d2f2600 Mar 16 03:17:36 john29 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:36 john28 kernel: Lustre: dagg-OST0002-osc-ffff8817d9cf4000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:17:36 john15 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff8821acb8b300 x1594448323939424/t0(0) o4->dagg-OST0002-osc-ffff882fde9f4800@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john15 kernel: Lustre: dagg-OST0002-osc-ffff882fde9f4800: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 arkle7 kernel: Lustre: dagg-OST000d: Client c2000ae7-b551-1f4a-4f30-41e5a9e31fb7 (at 192.168.44.129@o2ib44) reconnecting Mar 16 03:17:36 arkle7 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:36 arkle7 kernel: Lustre: dagg-OST000d: Connection restored to e260c649-9240-d4bd-eba4-8694f00df919 (at 192.168.44.129@o2ib44) Mar 16 03:17:36 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9767400 Mar 16 03:17:36 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9767400 Mar 16 03:17:36 arkle5 kernel: Lustre: dagg-OST0009: Client 0202600d-b066-c547-a639-ee53deb97148 (at 192.168.44.132@o2ib44) reconnecting Mar 16 03:17:36 arkle5 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:17:36 arkle5 kernel: Lustre: dagg-OST0009: Connection restored to ac18b8dd-5f9b-d1d9-555f-48ee1cd5b8ea (at 192.168.44.132@o2ib44) Mar 16 03:17:36 arkle7 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:36 arkle5 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:17:36 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881efe3b6200 Mar 16 03:17:36 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881efe3b6200 Mar 16 03:17:36 john15 kernel: Lustre: dagg-OST0002-osc-ffff882fde9f4800: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:17:36 john29 kernel: Lustre: dagg-OST000d-osc-ffff88189afee000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:36 john29 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:36 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed81600 Mar 16 03:17:36 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed81600 Mar 16 03:17:36 john32 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff880839e29b00 x1594699396465568/t0(0) o4->dagg-OST0006-osc-ffff8817c5384000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john32 kernel: Lustre: dagg-OST0009-osc-ffff8817c5384000: Connection to dagg-OST0009 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 john32 kernel: Lustre: dagg-OST0009-osc-ffff8817c5384000: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:17:36 john32 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:36 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88091d4afe00 Mar 16 03:17:36 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88091d4afe00 Mar 16 03:17:36 arkle4 kernel: Lustre: dagg-OST0006: Client 0202600d-b066-c547-a639-ee53deb97148 (at 192.168.44.132@o2ib44) reconnecting Mar 16 03:17:36 arkle4 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:17:36 arkle4 kernel: Lustre: dagg-OST0006: Connection restored to ac18b8dd-5f9b-d1d9-555f-48ee1cd5b8ea (at 192.168.44.132@o2ib44) Mar 16 03:17:36 arkle4 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:17:36 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c65f1da00 Mar 16 03:17:36 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c65f1da00 Mar 16 03:17:36 john17 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff881a33aa1b00 x1594448316855856/t0(0) o4->dagg-OST0007-osc-ffff881899e5b000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:36 john17 kernel: Lustre: dagg-OST0007-osc-ffff881899e5b000: Connection to dagg-OST0007 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:36 john17 kernel: Lustre: dagg-OST0007-osc-ffff881899e5b000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:17:36 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e093a00 Mar 16 03:17:36 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e093a00 Mar 16 03:17:36 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881751ca3e00 Mar 16 03:17:36 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881751ca3e00 Mar 16 03:17:37 john31 kernel: Lustre: 914:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff880f36e4bf00 x1594699394169536/t0(0) o4->dagg-OST000b-osc-ffff882fde9fd000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130656 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:37 john31 kernel: Lustre: 914:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:37 john31 kernel: Lustre: dagg-OST000b-osc-ffff882fde9fd000: Connection to dagg-OST000b (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:37 john31 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:37 john31 kernel: Lustre: dagg-OST000b-osc-ffff882fde9fd000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:17:37 john31 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:37 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ed05eb200 Mar 16 03:17:37 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ed05eb200 Mar 16 03:17:37 john18 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130613/real 1521130613] req@ffff88262735fb00 x1594448284373008/t0(0) o4->dagg-OST0003-osc-ffff88015a243000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130657 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:37 john18 kernel: Lustre: dagg-OST0003-osc-ffff88015a243000: Connection to dagg-OST0003 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:37 arkle2 kernel: Lustre: dagg-OST0003: Client 5abb6bc1-b79c-bc3d-60a6-8103bee751c7 (at 192.168.44.118@o2ib44) reconnecting Mar 16 03:17:37 arkle2 kernel: Lustre: Skipped 7 previous similar messages Mar 16 03:17:37 arkle2 kernel: Lustre: dagg-OST0003: Connection restored to aad63549-8ffd-bbaa-adc4-c670c37277c9 (at 192.168.44.118@o2ib44) Mar 16 03:17:37 arkle2 kernel: Lustre: Skipped 7 previous similar messages Mar 16 03:17:37 john18 kernel: Lustre: dagg-OST0003-osc-ffff88015a243000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:17:37 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881efe3b6200 Mar 16 03:17:37 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881efe3b6200 Mar 16 03:17:37 john30 kernel: Lustre: 919:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130613/real 1521130613] req@ffff88172d3c6900 x1594699400487456/t0(0) o4->dagg-OST0005-osc-ffff88189ac92000@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130657 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:37 john30 kernel: Lustre: 919:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:17:37 john30 kernel: Lustre: dagg-OST0005-osc-ffff88189ac92000: Connection to dagg-OST0005 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:37 john30 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:37 arkle3 kernel: Lustre: dagg-OST0005: Client faed83b4-3a5e-2a8e-2882-7e99d174dfb0 (at 192.168.44.130@o2ib44) reconnecting Mar 16 03:17:37 arkle3 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:17:37 arkle3 kernel: Lustre: dagg-OST0005: Connection restored to 2beb6461-856d-b027-2da8-040c1c990221 (at 192.168.44.130@o2ib44) Mar 16 03:17:37 arkle3 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:17:37 john30 kernel: Lustre: dagg-OST0005-osc-ffff88189ac92000: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:17:37 john30 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:37 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dc52f1a00 Mar 16 03:17:37 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dc52f1a00 Mar 16 03:17:37 john12 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130613/real 1521130613] req@ffff880903513600 x1594448324468928/t0(0) o4->dagg-OST000c-osc-ffff88015a3ab800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130657 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:37 john12 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:37 john12 kernel: Lustre: dagg-OST000c-osc-ffff88015a3ab800: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:37 john12 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:37 john12 kernel: Lustre: dagg-OST000c-osc-ffff88015a3ab800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:37 john12 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:37 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881d6d5da400 Mar 16 03:17:37 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881d6d5da400 Mar 16 03:17:37 john6 kernel: Lustre: 948:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130613/real 1521130613] req@ffff882e21396300 x1594448323932400/t0(0) o4->dagg-OST0007-osc-ffff881899c73800@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130657 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:37 john6 kernel: Lustre: 948:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:37 john6 kernel: Lustre: dagg-OST0007-osc-ffff881899c73800: Connection to dagg-OST0007 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:37 john6 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:37 john6 kernel: Lustre: dagg-OST0007-osc-ffff881899c73800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:17:37 john6 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:37 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e093a00 Mar 16 03:17:37 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e093a00 Mar 16 03:17:37 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c65f1da00 Mar 16 03:17:37 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c65f1da00 Mar 16 03:17:37 john33 kernel: Lustre: 910:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130613/real 1521130613] req@ffff8814dc7e5100 x1594699393884960/t0(0) o4->dagg-OST000f-osc-ffff88006bb84800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130657 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:37 john33 kernel: Lustre: dagg-OST000f-osc-ffff88006bb84800: Connection to dagg-OST000f (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:37 arkle8 kernel: Lustre: dagg-OST000f: Connection restored to 6e809e98-956d-fb0a-6999-afce652c717c (at 192.168.44.133@o2ib44) Mar 16 03:17:37 john33 kernel: Lustre: dagg-OST000f-osc-ffff88006bb84800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:17:37 john23 kernel: Lustre: 950:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130612/real 1521130612] req@ffff8820366d6900 x1594448282177696/t0(0) o4->dagg-OST0009-osc-ffff882fdbc22800@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130657 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:37 john23 kernel: Lustre: dagg-OST0009-osc-ffff882fdbc22800: Connection to dagg-OST0009 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:37 john23 kernel: Lustre: dagg-OST0009-osc-ffff882fdbc22800: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:17:37 john16 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130613/real 1521130613] req@ffff880bbdede900 x1594448321508048/t0(0) o4->dagg-OST000c-osc-ffff882fc87e8000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130657 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:37 john16 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:37 john16 kernel: Lustre: dagg-OST000c-osc-ffff882fc87e8000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:37 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e1b697400 Mar 16 03:17:37 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e1b697400 Mar 16 03:17:37 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e1b697400 Mar 16 03:17:37 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e1b697400 Mar 16 03:17:37 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c65f1da00 Mar 16 03:17:37 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c65f1da00 Mar 16 03:17:37 john16 kernel: Lustre: dagg-OST000c-osc-ffff882fc87e8000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:37 john13 kernel: Lustre: 931:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130613/real 1521130613] req@ffff8801a13e6300 x1594448319201264/t0(0) o4->dagg-OST0000-osc-ffff88015a2c6800@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130657 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:37 john13 kernel: Lustre: 931:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:17:37 john13 kernel: Lustre: dagg-OST0000-osc-ffff88015a2c6800: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:37 john13 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:17:37 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88091d4afe00 Mar 16 03:17:37 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88091d4afe00 Mar 16 03:17:37 arkle1 kernel: Lustre: dagg-OST0000: Client 240b2d78-e95a-c921-2bc6-0bc7648161c4 (at 192.168.44.113@o2ib44) reconnecting Mar 16 03:17:37 arkle1 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:17:37 arkle1 kernel: Lustre: dagg-OST0000: Connection restored to 03efa853-fec8-8f3a-4bf6-da90da4da162 (at 192.168.44.113@o2ib44) Mar 16 03:17:38 arkle1 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:17:38 john10 kernel: Lustre: 943:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130613/real 1521130613] req@ffff882063fff800 x1594448319882256/t0(0) o4->dagg-OST0001-osc-ffff882fd4b83800@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130657 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:38 john10 kernel: Lustre: dagg-OST000f-osc-ffff882fd4b83800: Connection to dagg-OST000f (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:38 john10 kernel: Lustre: dagg-OST000f-osc-ffff882fd4b83800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:17:38 john10 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448319913808 length 1048576 too big: 1046464 left, 1046464 allowed Mar 16 03:17:38 john10 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:38 john10 kernel: Lustre: 943:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:38 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808913ff600 Mar 16 03:17:38 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808913ff600 Mar 16 03:17:38 john13 kernel: Lustre: dagg-OST0000-osc-ffff88015a2c6800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:17:38 john13 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:17:38 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88263d2f4800 Mar 16 03:17:38 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88263d2f4800 Mar 16 03:17:38 john13 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448319243072 length 1048576 too big: 1048356 left, 1048356 allowed Mar 16 03:17:38 john13 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:17:38 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cb200 Mar 16 03:17:38 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cb200 Mar 16 03:17:38 john26 kernel: Lustre: 929:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130614/real 1521130614] req@ffff881cd7d1e300 x1594448281333600/t0(0) o4->dagg-OST0000-osc-ffff882fde832000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130658 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:38 john26 kernel: Lustre: dagg-OST0000-osc-ffff882fde832000: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:38 john26 kernel: Lustre: dagg-OST0000-osc-ffff882fde832000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:17:38 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cb200 Mar 16 03:17:38 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cb200 Mar 16 03:17:38 john14 kernel: Lustre: 929:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130614/real 1521130614] req@ffff880beba30f00 x1594448317088528/t0(0) o4->dagg-OST000b-osc-ffff88189a368800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130658 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:38 john14 kernel: Lustre: dagg-OST000b-osc-ffff88189a368800: Connection to dagg-OST000b (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:38 john14 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:38 arkle6 kernel: Lustre: dagg-OST000b: Client 23017e25-beb4-2d0c-1e7b-9a1bd7cbb415 (at 192.168.44.114@o2ib44) reconnecting Mar 16 03:17:38 arkle6 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:38 arkle6 kernel: Lustre: dagg-OST000b: Connection restored to 78033fff-f4e3-2589-3920-d6d5c19d59a2 (at 192.168.44.114@o2ib44) Mar 16 03:17:38 john32 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130614/real 1521130614] req@ffff882c89e68600 x1594699396468656/t0(0) o4->dagg-OST0007-osc-ffff8817c5384000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130658 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:38 john32 kernel: Lustre: dagg-OST0007-osc-ffff8817c5384000: Connection to dagg-OST0007 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:38 john32 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:38 arkle6 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:38 john32 kernel: Lustre: dagg-OST0007-osc-ffff8817c5384000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:17:38 john32 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:38 john14 kernel: Lustre: dagg-OST000b-osc-ffff88189a368800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:17:38 john14 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:38 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e093a00 Mar 16 03:17:38 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e093a00 Mar 16 03:17:38 arkle4 kernel: LustreError: 187185:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8803133d5050 x1594699396468656/t0(0) o4->0202600d-b066-c547-a639-ee53deb97148@192.168.44.132@o2ib44:491/0 lens 608/448 e 0 to 0 dl 1521130701 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:17:38 john14 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448317128112 length 1048576 too big: 1048504 left, 1048504 allowed Mar 16 03:17:38 john14 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 2 previous similar messages Mar 16 03:17:38 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880052560a00 Mar 16 03:17:38 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880052560a00 Mar 16 03:17:38 arkle4 kernel: LustreError: 187185:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 13 previous similar messages Mar 16 03:17:38 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ae0cee000 Mar 16 03:17:38 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ae0cee000 Mar 16 03:17:38 arkle4 kernel: Lustre: dagg-OST0007: Bulk IO write error with 0202600d-b066-c547-a639-ee53deb97148 (at 192.168.44.132@o2ib44), client will retry: rc = -110 Mar 16 03:17:38 arkle4 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:17:38 john28 kernel: Lustre: 918:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130614/real 1521130614] req@ffff880a09adb900 x1594448282256576/t0(0) o4->dagg-OST000b-osc-ffff8817d9cf4000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130658 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:38 john28 kernel: Lustre: dagg-OST000b-osc-ffff8817d9cf4000: Connection to dagg-OST000b (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:38 john28 kernel: Lustre: dagg-OST000b-osc-ffff8817d9cf4000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:17:38 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882df6edd000 Mar 16 03:17:38 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882df6edd000 Mar 16 03:17:38 arkle4 kernel: Lustre: dagg-OST0007: Client 22c507bc-bb1f-dbcf-8316-529c1039bbfe (at 192.168.44.133@o2ib44) reconnecting Mar 16 03:17:38 arkle4 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:17:38 arkle4 kernel: Lustre: dagg-OST0007: Connection restored to 6e809e98-956d-fb0a-6999-afce652c717c (at 192.168.44.133@o2ib44) Mar 16 03:17:38 arkle4 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:17:38 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8805d2a9ea00 Mar 16 03:17:38 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8805d2a9ea00 Mar 16 03:17:38 john8 kernel: Lustre: 942:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130614/real 1521130614] req@ffff882770e66600 x1594448316809648/t0(0) o4->dagg-OST000f-osc-ffff88015a2fe000@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130658 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:38 john8 kernel: Lustre: dagg-OST000f-osc-ffff88015a2fe000: Connection to dagg-OST000f (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:39 john8 kernel: Lustre: dagg-OST000f-osc-ffff88015a2fe000: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:17:39 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880052560a00 Mar 16 03:17:39 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880052560a00 Mar 16 03:17:39 john6 kernel: Lustre: 953:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130615/real 1521130615] req@ffff8819dca5fb00 x1594448323934928/t0(0) o4->dagg-OST0009-osc-ffff881899c73800@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130659 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:39 john6 kernel: Lustre: 953:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:39 john6 kernel: Lustre: dagg-OST0009-osc-ffff881899c73800: Connection to dagg-OST0009 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:39 john6 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:39 arkle5 kernel: Lustre: dagg-OST0009: Client efbad07f-625e-4fc5-c94b-ef8442fed4fb (at 192.168.44.106@o2ib44) reconnecting Mar 16 03:17:39 arkle5 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:17:39 arkle5 kernel: Lustre: dagg-OST0009: Connection restored to 3c1483cd-dc70-d4a2-d9df-82049cf8a62d (at 192.168.44.106@o2ib44) Mar 16 03:17:39 arkle5 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:17:39 john6 kernel: Lustre: dagg-OST0009-osc-ffff881899c73800: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:17:39 john6 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:39 john6 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594448323977504 length 1048576 too big: 1046332 left, 1046332 allowed Mar 16 03:17:39 john6 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:17:39 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed82600 Mar 16 03:17:39 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed82600 Mar 16 03:17:39 john7 kernel: Lustre: 951:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130615/real 1521130615] req@ffff881951f70c00 x1594448319410432/t0(0) o4->dagg-OST000b-osc-ffff882fda4b4000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130659 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:39 john7 kernel: Lustre: 951:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:39 john7 kernel: Lustre: dagg-OST000b-osc-ffff882fda4b4000: Connection to dagg-OST000b (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:39 john7 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:39 john7 kernel: Lustre: dagg-OST000b-osc-ffff882fda4b4000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:17:39 john7 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:39 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882df6edc800 Mar 16 03:17:39 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882df6edc800 Mar 16 03:17:40 john18 kernel: Lustre: 923:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130615/real 1521130615] req@ffff882c3a92d700 x1594448284393072/t0(0) o4->dagg-OST0001-osc-ffff88015a243000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130659 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:40 john18 kernel: Lustre: dagg-OST0001-osc-ffff88015a243000: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:40 arkle1 kernel: Lustre: dagg-OST0001: Client 5abb6bc1-b79c-bc3d-60a6-8103bee751c7 (at 192.168.44.118@o2ib44) reconnecting Mar 16 03:17:40 arkle1 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:40 arkle1 kernel: Lustre: dagg-OST0001: Connection restored to aad63549-8ffd-bbaa-adc4-c670c37277c9 (at 192.168.44.118@o2ib44) Mar 16 03:17:40 arkle1 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:40 john18 kernel: Lustre: dagg-OST0001-osc-ffff88015a243000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:17:40 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88263d2f4800 Mar 16 03:17:40 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88263d2f4800 Mar 16 03:17:40 john16 kernel: Lustre: 941:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130616/real 1521130616] req@ffff881e67307200 x1594448321512432/t0(0) o4->dagg-OST000d-osc-ffff882fc87e8000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130660 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:40 john16 kernel: Lustre: dagg-OST000d-osc-ffff882fc87e8000: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:40 arkle7 kernel: Lustre: dagg-OST000d: Client 555ab62f-a8a1-c567-5cc2-8766e0a20f71 (at 192.168.44.116@o2ib44) reconnecting Mar 16 03:17:40 arkle7 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:40 arkle7 kernel: Lustre: dagg-OST000d: Connection restored to 383277e6-1fbe-5568-636a-5a9740cb6a43 (at 192.168.44.116@o2ib44) Mar 16 03:17:40 arkle7 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:40 john16 kernel: Lustre: dagg-OST000d-osc-ffff882fc87e8000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:40 arkle2 kernel: Lustre: dagg-OST0003: Client 0ec03030-04bc-e99c-053e-6e02b04218ff (at 192.168.44.128@o2ib44) reconnecting Mar 16 03:17:40 arkle2 kernel: Lustre: dagg-OST0003: Connection restored to 82268709-2c1f-0fd2-7b7f-5ddbcb19b9e0 (at 192.168.44.128@o2ib44) Mar 16 03:17:40 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88176883d800 Mar 16 03:17:40 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88176883d800 Mar 16 03:17:40 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88091d4afe00 Mar 16 03:17:40 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88091d4afe00 Mar 16 03:17:40 john17 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130615/real 1521130615] req@ffff880d6a549200 x1594448316861280/t0(0) o4->dagg-OST0009-osc-ffff881899e5b000@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130660 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:40 john17 kernel: Lustre: dagg-OST0009-osc-ffff881899e5b000: Connection to dagg-OST0009 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:40 john17 kernel: Lustre: dagg-OST0009-osc-ffff881899e5b000: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:17:40 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c65f1da00 Mar 16 03:17:40 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c65f1da00 Mar 16 03:17:40 john17 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594448316904944 length 1048576 too big: 1047576 left, 1047576 allowed Mar 16 03:17:40 john17 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:17:40 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88221bdcf400 Mar 16 03:17:40 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88221bdcf400 Mar 16 03:17:41 john10 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130616/real 1521130616] req@ffff88100c6d1e00 x1594448319889536/t0(0) o4->dagg-OST0004-osc-ffff882fd4b83800@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130660 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:41 john10 kernel: Lustre: dagg-OST0008-osc-ffff882fd4b83800: Connection to dagg-OST0008 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:41 john10 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:41 john10 kernel: Lustre: dagg-OST0008-osc-ffff882fd4b83800: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:17:41 john10 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:41 john10 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:41 arkle3 kernel: Lustre: dagg-OST0004: Client 642745b2-465c-4997-8a20-40d9249d2ad8 (at 192.168.44.110@o2ib44) reconnecting Mar 16 03:17:41 arkle3 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:41 arkle3 kernel: Lustre: dagg-OST0004: Connection restored to 5f2c6b52-0a2a-c332-fc42-ab2c0ecf1f9e (at 192.168.44.110@o2ib44) Mar 16 03:17:41 arkle3 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:41 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dc52f1200 Mar 16 03:17:41 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dc52f1200 Mar 16 03:17:41 john7 kernel: Lustre: 923:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130617/real 1521130617] req@ffff8817a06fb600 x1594448319411968/t0(0) o4->dagg-OST000c-osc-ffff882fda4b4000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130661 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:41 john7 kernel: Lustre: dagg-OST000c-osc-ffff882fda4b4000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:41 john7 kernel: Lustre: dagg-OST000c-osc-ffff882fda4b4000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:41 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881d6d5d8c00 Mar 16 03:17:41 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881d6d5d8c00 Mar 16 03:17:42 john23 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130618/real 1521130618] req@ffff882030f12100 x1594448282185456/t0(0) o4->dagg-OST000b-osc-ffff882fdbc22800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130662 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:42 john23 kernel: Lustre: dagg-OST000b-osc-ffff882fdbc22800: Connection to dagg-OST000b (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:42 arkle6 kernel: Lustre: dagg-OST000b: Client e5e03fa1-aa9e-78b9-d06c-6e18b34089cf (at 192.168.44.123@o2ib44) reconnecting Mar 16 03:17:42 arkle6 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:42 arkle6 kernel: Lustre: dagg-OST000b: Connection restored to 1fea7452-ea20-7788-5be4-f826c8c7f864 (at 192.168.44.123@o2ib44) Mar 16 03:17:42 arkle6 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:42 john9 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130618/real 1521130618] req@ffff8822bf6a1b00 x1594448323976576/t0(0) o4->dagg-OST000a-osc-ffff882fdd33b800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130662 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:42 john9 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:42 john9 kernel: Lustre: dagg-OST000a-osc-ffff882fdd33b800: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:42 john9 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:42 john9 kernel: Lustre: dagg-OST000a-osc-ffff882fdd33b800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:17:42 john9 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:42 john23 kernel: Lustre: dagg-OST000b-osc-ffff882fdbc22800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:17:42 john23 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448282227488 length 1048576 too big: 1046120 left, 1046120 allowed Mar 16 03:17:42 john23 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 2 previous similar messages Mar 16 03:17:42 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882d8ea71000 Mar 16 03:17:42 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882d8ea71000 Mar 16 03:17:42 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882df6edee00 Mar 16 03:17:42 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882df6edee00 Mar 16 03:17:42 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882df6edee00 Mar 16 03:17:42 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882df6edee00 Mar 16 03:17:43 john17 kernel: Lustre: 956:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130618/real 1521130618] req@ffff8824f3bcb000 x1594448316863856/t0(0) o4->dagg-OST0003-osc-ffff881899e5b000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130662 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:43 john17 kernel: Lustre: dagg-OST0003-osc-ffff881899e5b000: Connection to dagg-OST0003 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:43 john17 kernel: Lustre: dagg-OST0003-osc-ffff881899e5b000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:17:43 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881768838200 Mar 16 03:17:43 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881768838200 Mar 16 03:17:43 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881d6d5de600 Mar 16 03:17:43 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881d6d5de600 Mar 16 03:17:43 john22 kernel: Lustre: 931:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130619/real 1521130619] req@ffff880c93ec4e00 x1594448280693824/t0(0) o4->dagg-OST0001-osc-ffff88189a897000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130663 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:43 john22 kernel: Lustre: 931:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:43 john22 kernel: Lustre: dagg-OST0001-osc-ffff88189a897000: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:43 john22 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:43 john22 kernel: Lustre: dagg-OST0001-osc-ffff88189a897000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:17:43 john22 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:43 john22 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448280717776 length 1048576 too big: 1047940 left, 1047940 allowed Mar 16 03:17:43 john22 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 2 previous similar messages Mar 16 03:17:43 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cb200 Mar 16 03:17:43 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cb200 Mar 16 03:17:43 arkle1 kernel: LustreError: 18036:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88175c7f9c50 x1594448280693824/t0(0) o4->53090394-a3ef-6106-82db-77d8ecba9e98@192.168.44.122@o2ib44:496/0 lens 608/448 e 0 to 0 dl 1521130706 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:17:43 arkle1 kernel: LustreError: 18036:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 12 previous similar messages Mar 16 03:17:43 arkle1 kernel: Lustre: dagg-OST0001: Bulk IO write error with 53090394-a3ef-6106-82db-77d8ecba9e98 (at 192.168.44.122@o2ib44), client will retry: rc = -110 Mar 16 03:17:43 arkle1 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:17:44 john28 kernel: Lustre: 936:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130620/real 1521130620] req@ffff8819a6e74200 x1594448282279632/t0(0) o4->dagg-OST0005-osc-ffff8817d9cf4000@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130664 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:44 john28 kernel: Lustre: 936:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:44 john28 kernel: Lustre: dagg-OST0005-osc-ffff8817d9cf4000: Connection to dagg-OST0005 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:44 john28 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:44 john28 kernel: Lustre: dagg-OST0005-osc-ffff8817d9cf4000: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:17:44 john28 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:44 john28 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448282304224 length 1048576 too big: 1046944 left, 1046944 allowed Mar 16 03:17:44 john28 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:17:44 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dc52f1200 Mar 16 03:17:44 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dc52f1200 Mar 16 03:17:46 john16 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130621/real 1521130621] req@ffff882adee1a400 x1594448321516688/t0(0) o4->dagg-OST0009-osc-ffff882fc87e8000@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130666 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:46 john16 kernel: Lustre: dagg-OST0009-osc-ffff882fc87e8000: Connection to dagg-OST0009 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:46 arkle5 kernel: Lustre: dagg-OST0009: Client 555ab62f-a8a1-c567-5cc2-8766e0a20f71 (at 192.168.44.116@o2ib44) reconnecting Mar 16 03:17:46 arkle5 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:46 arkle5 kernel: Lustre: dagg-OST0009: Connection restored to 383277e6-1fbe-5568-636a-5a9740cb6a43 (at 192.168.44.116@o2ib44) Mar 16 03:17:46 arkle5 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:46 john16 kernel: Lustre: dagg-OST0009-osc-ffff882fc87e8000: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:17:46 john16 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594448321541088 length 1048576 too big: 1046776 left, 1046776 allowed Mar 16 03:17:46 john16 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:17:46 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed83400 Mar 16 03:17:46 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed83400 Mar 16 03:17:46 arkle5 kernel: LustreError: 449866:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881619b07850 x1594448321516688/t0(0) o4->555ab62f-a8a1-c567-5cc2-8766e0a20f71@192.168.44.116@o2ib44:499/0 lens 608/448 e 0 to 0 dl 1521130709 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:17:46 arkle5 kernel: LustreError: 449866:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 14 previous similar messages Mar 16 03:17:46 arkle5 kernel: Lustre: dagg-OST0009: Bulk IO write error with 555ab62f-a8a1-c567-5cc2-8766e0a20f71 (at 192.168.44.116@o2ib44), client will retry: rc = -110 Mar 16 03:17:46 arkle5 kernel: Lustre: Skipped 14 previous similar messages Mar 16 03:17:47 john19 kernel: Lustre: 908:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130623/real 1521130623] req@ffff881d29064b00 x1594448327445664/t0(0) o4->dagg-OST000d-osc-ffff88189a323000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130667 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:47 john19 kernel: Lustre: dagg-OST000d-osc-ffff88189a323000: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:47 arkle7 kernel: Lustre: dagg-OST000d: Client d4acd478-18fd-71d3-c177-089c607f8639 (at 192.168.44.119@o2ib44) reconnecting Mar 16 03:17:47 arkle7 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:47 arkle7 kernel: Lustre: dagg-OST000d: Connection restored to e1a18b29-346e-ff08-5424-24003125f711 (at 192.168.44.119@o2ib44) Mar 16 03:17:47 arkle7 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:47 john19 kernel: Lustre: dagg-OST000d-osc-ffff88189a323000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:17:47 john19 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448327487568 length 1048576 too big: 1044904 left, 1044904 allowed Mar 16 03:17:47 john19 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 2 previous similar messages Mar 16 03:17:47 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88091d4aea00 Mar 16 03:17:47 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88091d4aea00 Mar 16 03:17:55 john26 kernel: Lustre: 942:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130631/real 1521130631] req@ffff881dfc3ff800 x1594448281339792/t0(0) o4->dagg-OST000e-osc-ffff882fde832000@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130675 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:55 john26 kernel: Lustre: dagg-OST000e-osc-ffff882fde832000: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:55 arkle8 kernel: Lustre: dagg-OST000e: Client f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44) reconnecting Mar 16 03:17:55 arkle8 kernel: Lustre: Skipped 6 previous similar messages Mar 16 03:17:55 arkle8 kernel: Lustre: dagg-OST000e: Connection restored to d5de0e2d-4431-41fa-738a-2fa7d0a3c365 (at 192.168.44.126@o2ib44) Mar 16 03:17:55 arkle8 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:17:55 john26 kernel: Lustre: dagg-OST000e-osc-ffff882fde832000: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:17:55 john26 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448281384464 length 1048576 too big: 1047960 left, 1047960 allowed Mar 16 03:17:55 john26 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 2 previous similar messages Mar 16 03:17:55 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8810060a6c00 Mar 16 03:17:55 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8810060a6c00 Mar 16 03:17:55 arkle8 kernel: LustreError: 303900:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff880c6c73dc50 x1594448281339792/t0(0) o4->f4a3cf53-7286-0d27-5cb4-5f6321872642@192.168.44.126@o2ib44:508/0 lens 608/448 e 0 to 0 dl 1521130718 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:17:55 arkle8 kernel: LustreError: 303900:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 6 previous similar messages Mar 16 03:17:55 arkle8 kernel: Lustre: dagg-OST000e: Bulk IO write error with f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44), client will retry: rc = -110 Mar 16 03:17:55 arkle8 kernel: Lustre: Skipped 6 previous similar messages Mar 16 03:17:55 john11 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130631/real 1521130631] req@ffff8826ea14ad00 x1594448320168128/t0(0) o4->dagg-OST000a-osc-ffff882fd571f000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130675 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:55 john11 kernel: Lustre: dagg-OST000a-osc-ffff882fd571f000: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:55 arkle6 kernel: Lustre: dagg-OST000a: Client 7c2b1b78-121a-b208-fc8a-57295d42d7f7 (at 192.168.44.111@o2ib44) reconnecting Mar 16 03:17:55 arkle6 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:55 arkle6 kernel: Lustre: dagg-OST000a: Connection restored to 9e90df21-9b59-537c-f486-0efeb807faad (at 192.168.44.111@o2ib44) Mar 16 03:17:55 arkle6 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:17:55 john11 kernel: Lustre: dagg-OST000a-osc-ffff882fd571f000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:17:55 john11 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448320226160 length 1048576 too big: 1047284 left, 1047284 allowed Mar 16 03:17:55 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ae0ce9e00 Mar 16 03:17:55 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ae0ce9e00 Mar 16 03:17:55 arkle6 kernel: LustreError: 330393:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88101af96050 x1594448320168128/t0(0) o4->7c2b1b78-121a-b208-fc8a-57295d42d7f7@192.168.44.111@o2ib44:508/0 lens 608/448 e 0 to 0 dl 1521130718 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:17:55 arkle6 kernel: LustreError: 330393:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 14 previous similar messages Mar 16 03:17:55 arkle6 kernel: Lustre: dagg-OST000a: Bulk IO write error with 7c2b1b78-121a-b208-fc8a-57295d42d7f7 (at 192.168.44.111@o2ib44), client will retry: rc = -110 Mar 16 03:17:55 arkle6 kernel: Lustre: Skipped 14 previous similar messages Mar 16 03:17:55 john25 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130631/real 1521130631] req@ffff8824baa20f00 x1594448280895840/t0(0) o4->dagg-OST000e-osc-ffff882fd7db5800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130675 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:55 john25 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:17:55 john25 kernel: Lustre: dagg-OST000e-osc-ffff882fd7db5800: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:55 john25 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:55 john25 kernel: Lustre: dagg-OST000e-osc-ffff882fd7db5800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:17:55 john25 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:55 john25 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448280923264 length 1048576 too big: 1046340 left, 1046340 allowed Mar 16 03:17:55 john25 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:17:55 arkle8 kernel: LustreError: 298958:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e1b695c00 Mar 16 03:17:55 arkle8 kernel: LustreError: 298958:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e1b695c00 Mar 16 03:17:56 john12 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130632/real 1521130632] req@ffff881f017c1800 x1594448324471616/t0(0) o4->dagg-OST000e-osc-ffff88015a3ab800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130676 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:56 john12 kernel: Lustre: dagg-OST000e-osc-ffff88015a3ab800: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:56 john12 kernel: Lustre: dagg-OST000e-osc-ffff88015a3ab800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:17:56 john12 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448324516144 length 1048576 too big: 1046984 left, 1046984 allowed Mar 16 03:17:56 john12 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 2 previous similar messages Mar 16 03:17:56 arkle8 kernel: LustreError: 298958:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e1b694200 Mar 16 03:17:56 arkle8 kernel: LustreError: 298958:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e1b694200 Mar 16 03:17:56 john31 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130632/real 1521130632] req@ffff881c56f45100 x1594699394190000/t0(0) o4->dagg-OST000e-osc-ffff882fde9fd000@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130676 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:56 john31 kernel: Lustre: dagg-OST000e-osc-ffff882fde9fd000: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:56 john31 kernel: Lustre: dagg-OST000e-osc-ffff882fde9fd000: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:17:56 john31 kernel: LNetError: 897:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594699394217600 length 1048576 too big: 1048208 left, 1048208 allowed Mar 16 03:17:56 john31 kernel: LNetError: 897:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 2 previous similar messages Mar 16 03:17:56 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880941095e00 Mar 16 03:17:56 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880941095e00 Mar 16 03:17:57 john11 kernel: Lustre: 931:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130633/real 1521130633] req@ffff8826ea148000 x1594448320175040/t0(0) o4->dagg-OST000e-osc-ffff882fd571f000@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130677 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:57 john11 kernel: Lustre: dagg-OST000e-osc-ffff882fd571f000: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:57 john11 kernel: Lustre: dagg-OST000e-osc-ffff882fd571f000: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:17:57 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8812b28a4800 Mar 16 03:17:57 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8812b28a4800 Mar 16 03:17:57 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e44058a00 Mar 16 03:17:57 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e44058a00 Mar 16 03:17:57 john15 kernel: Lustre: 919:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130633/real 1521130633] req@ffff880e86ebb300 x1594448323943024/t0(0) o4->dagg-OST000e-osc-ffff882fde9f4800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130677 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:57 john15 kernel: Lustre: dagg-OST000e-osc-ffff882fde9f4800: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:57 john15 kernel: Lustre: dagg-OST000e-osc-ffff882fde9f4800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:17:57 john15 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448323987536 length 1048576 too big: 1046392 left, 1046392 allowed Mar 16 03:17:57 john15 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 2 previous similar messages Mar 16 03:17:57 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880941095e00 Mar 16 03:17:57 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880941095e00 Mar 16 03:17:58 john11 kernel: Lustre: 918:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130634/real 1521130634] req@ffff880940a70000 x1594448320177872/t0(0) o4->dagg-OST0004-osc-ffff882fd571f000@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130678 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:17:58 john11 kernel: Lustre: dagg-OST0004-osc-ffff882fd571f000: Connection to dagg-OST0004 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:17:58 arkle3 kernel: Lustre: dagg-OST0004: Client 7c2b1b78-121a-b208-fc8a-57295d42d7f7 (at 192.168.44.111@o2ib44) reconnecting Mar 16 03:17:58 arkle3 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:58 arkle3 kernel: Lustre: dagg-OST0004: Connection restored to 9e90df21-9b59-537c-f486-0efeb807faad (at 192.168.44.111@o2ib44) Mar 16 03:17:58 arkle3 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:17:58 john11 kernel: Lustre: dagg-OST0004-osc-ffff882fd571f000: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:17:58 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880055643400 Mar 16 03:17:58 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880055643400 Mar 16 03:18:18 john25 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130654/real 1521130654] req@ffff881574af9e00 x1594448280872016/t0(0) o4->dagg-OST0006-osc-ffff882fd7db5800@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130698 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:18 john25 kernel: Lustre: dagg-OST0006-osc-ffff882fd7db5800: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:18 arkle4 kernel: Lustre: dagg-OST0006: Client 40d9b451-fab3-be40-89fb-0c9b04087671 (at 192.168.44.125@o2ib44) reconnecting Mar 16 03:18:18 arkle4 kernel: Lustre: dagg-OST0006: Connection restored to edb2edda-7927-928c-4497-0706e1cfa6ce (at 192.168.44.125@o2ib44) Mar 16 03:18:18 john25 kernel: Lustre: dagg-OST0006-osc-ffff882fd7db5800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:18:18 john25 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448280942512 length 1048576 too big: 1048092 left, 1048092 allowed Mar 16 03:18:18 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8801ebc89e00 Mar 16 03:18:18 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8801ebc89e00 Mar 16 03:18:18 john14 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130654/real 1521130654] req@ffff880a99ed1200 x1594448317087008/t0(0) o4->dagg-OST0006-osc-ffff88189a368800@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130698 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:18:18 john14 kernel: Lustre: dagg-OST0006-osc-ffff88189a368800: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:18 john30 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130654/real 1521130654] req@ffff882e4338a400 x1594699400481984/t0(0) o4->dagg-OST000c-osc-ffff88189ac92000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130698 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:18 john30 kernel: Lustre: dagg-OST000c-osc-ffff88189ac92000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:18 arkle7 kernel: Lustre: dagg-OST000c: Client faed83b4-3a5e-2a8e-2882-7e99d174dfb0 (at 192.168.44.130@o2ib44) reconnecting Mar 16 03:18:18 arkle7 kernel: Lustre: dagg-OST000c: Connection restored to 2beb6461-856d-b027-2da8-040c1c990221 (at 192.168.44.130@o2ib44) Mar 16 03:18:18 john14 kernel: Lustre: dagg-OST0006-osc-ffff88189a368800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:18:18 john14 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448317168624 length 1048576 too big: 1047896 left, 1047896 allowed Mar 16 03:18:18 john30 kernel: Lustre: dagg-OST000c-osc-ffff88189ac92000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:18:18 john30 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594699400552016 length 1048576 too big: 1048516 left, 1048516 allowed Mar 16 03:18:18 arkle4 kernel: LustreError: 302876:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882ddb73ec50 x1594448280872016/t0(0) o4->40d9b451-fab3-be40-89fb-0c9b04087671@192.168.44.125@o2ib44:531/0 lens 608/448 e 0 to 0 dl 1521130741 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:18:18 arkle4 kernel: LustreError: 302876:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 1 previous similar message Mar 16 03:18:18 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e097a00 Mar 16 03:18:18 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e097a00 Mar 16 03:18:18 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8ce00 Mar 16 03:18:18 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8ce00 Mar 16 03:18:18 arkle4 kernel: Lustre: dagg-OST0006: Bulk IO write error with 40d9b451-fab3-be40-89fb-0c9b04087671 (at 192.168.44.125@o2ib44), client will retry: rc = -110 Mar 16 03:18:18 john30 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:18:18 arkle4 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:18 john27 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130654/real 1521130654] req@ffff881b4e2d3600 x1594448285369712/t0(0) o4->dagg-OST000f-osc-ffff88015a1e4800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130698 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:18 john27 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:18:18 john27 kernel: Lustre: dagg-OST000f-osc-ffff88015a1e4800: Connection to dagg-OST000f (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:18 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881775c82e00 Mar 16 03:18:18 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881775c82e00 Mar 16 03:18:18 arkle7 kernel: LustreError: 34887:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882f03d42c50 x1594699400481984/t0(0) o4->faed83b4-3a5e-2a8e-2882-7e99d174dfb0@192.168.44.130@o2ib44:531/0 lens 608/448 e 0 to 0 dl 1521130741 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:18:18 john28 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130654/real 1521130654] req@ffff881aaab92d00 x1594448282246752/t0(0) o4->dagg-OST0006-osc-ffff8817d9cf4000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130698 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:18 john28 kernel: Lustre: dagg-OST0006-osc-ffff8817d9cf4000: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:18 john28 kernel: Lustre: dagg-OST0006-osc-ffff8817d9cf4000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:18:18 john27 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:18 john28 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448282327408 length 1048576 too big: 1045344 left, 1045344 allowed Mar 16 03:18:18 arkle7 kernel: LustreError: 34887:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 15 previous similar messages Mar 16 03:18:18 arkle8 kernel: Lustre: dagg-OST000f: Connection restored to c8b7715d-04bb-a76a-3db9-c45127314e23 (at 192.168.44.127@o2ib44) Mar 16 03:18:18 arkle8 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:18:18 arkle7 kernel: Lustre: dagg-OST000c: Bulk IO write error with faed83b4-3a5e-2a8e-2882-7e99d174dfb0 (at 192.168.44.130@o2ib44), client will retry: rc = -110 Mar 16 03:18:18 arkle7 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:18:18 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882efa9a9e00 Mar 16 03:18:18 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882efa9a9e00 Mar 16 03:18:18 john27 kernel: Lustre: dagg-OST000f-osc-ffff88015a1e4800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:18:18 john27 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:18 john20 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130654/real 1521130654] req@ffff8828a2b52100 x1594448327525968/t0(0) o4->dagg-OST000d-osc-ffff8817da6ad800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130698 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:18 john20 kernel: Lustre: dagg-OST000d-osc-ffff8817da6ad800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:18 john27 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448285439152 length 1048576 too big: 1046096 left, 1046096 allowed Mar 16 03:18:18 john27 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:18:18 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e44058800 Mar 16 03:18:18 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e44058800 Mar 16 03:18:18 john20 kernel: Lustre: dagg-OST000d-osc-ffff8817da6ad800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:18:18 john20 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448327628320 length 1048576 too big: 1045364 left, 1045364 allowed Mar 16 03:18:18 john20 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:18:18 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882eb988d400 Mar 16 03:18:18 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882eb988d400 Mar 16 03:18:18 john21 kernel: Lustre: 918:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130654/real 1521130654] req@ffff8813c2e68300 x1594448332914000/t0(0) o4->dagg-OST000c-osc-ffff882fc5588800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130698 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:18 john21 kernel: Lustre: 918:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:18:18 john21 kernel: Lustre: dagg-OST000c-osc-ffff882fc5588800: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:18 john21 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:18 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88091d4aa000 Mar 16 03:18:18 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88091d4aa000 Mar 16 03:18:18 john21 kernel: Lustre: dagg-OST000c-osc-ffff882fc5588800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:18:18 john21 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:18 john21 kernel: LNetError: 895:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448332969472 length 1048576 too big: 1044712 left, 1044712 allowed Mar 16 03:18:18 john21 kernel: LNetError: 895:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:18:18 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88091d4aa000 Mar 16 03:18:18 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88091d4aa000 Mar 16 03:18:18 john15 kernel: Lustre: 949:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130654/real 1521130654] req@ffff88207ef11200 x1594448323936064/t0(0) o4->dagg-OST000d-osc-ffff882fde9f4800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130698 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:18 john15 kernel: Lustre: dagg-OST000d-osc-ffff882fde9f4800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:18 john15 kernel: Lustre: dagg-OST000d-osc-ffff882fde9f4800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:18:18 john15 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448324006496 length 1048576 too big: 1045944 left, 1045944 allowed Mar 16 03:18:18 john24 kernel: Lustre: 951:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130654/real 1521130654] req@ffff882c67f3f500 x1594448284013248/t0(0) o4->dagg-OST000c-osc-ffff882fcb645000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130698 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:18 john24 kernel: Lustre: dagg-OST000c-osc-ffff882fcb645000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:18 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f0a18c600 Mar 16 03:18:18 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f0a18c600 Mar 16 03:18:18 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881e30d83c00 Mar 16 03:18:18 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881e30d83c00 Mar 16 03:18:18 john22 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130654/real 1521130654] req@ffff8823b403b900 x1594448280685024/t0(0) o4->dagg-OST000c-osc-ffff88189a897000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130698 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:18 john22 kernel: Lustre: dagg-OST000c-osc-ffff88189a897000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:18 john22 kernel: Lustre: dagg-OST000c-osc-ffff88189a897000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:18:18 john7 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130654/real 1521130654] req@ffff8817a06ff500 x1594448319400640/t0(0) o4->dagg-OST0006-osc-ffff882fda4b4000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130698 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:18 john7 kernel: Lustre: dagg-OST0006-osc-ffff882fda4b4000: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:18 john7 kernel: Lustre: dagg-OST0006-osc-ffff882fda4b4000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:18:18 john24 kernel: Lustre: dagg-OST000c-osc-ffff882fcb645000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:18:18 john24 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448284084096 length 1048576 too big: 1046748 left, 1046748 allowed Mar 16 03:18:18 john24 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:18:18 john22 kernel: LNetError: 904:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448280757904 length 1048576 too big: 1048436 left, 1048436 allowed Mar 16 03:18:19 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e4405b600 Mar 16 03:18:19 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e22509200 Mar 16 03:18:19 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e22509200 Mar 16 03:18:19 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e4405b600 Mar 16 03:18:19 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881e30d83c00 Mar 16 03:18:19 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881e30d83c00 Mar 16 03:18:18 john7 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448319477056 length 1048576 too big: 1045464 left, 1045464 allowed Mar 16 03:18:19 john7 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:18:19 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e21a6fe00 Mar 16 03:18:19 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e21a6fe00 Mar 16 03:18:19 john17 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff880e00b31500 x1594448316853392/t0(0) o4->dagg-OST0002-osc-ffff881899e5b000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:19 john17 kernel: Lustre: dagg-OST0002-osc-ffff881899e5b000: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 arkle2 kernel: Lustre: dagg-OST0002: Client 6b4209e2-620e-030d-e2d1-b9c575162077 (at 192.168.44.117@o2ib44) reconnecting Mar 16 03:18:19 arkle2 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:19 arkle2 kernel: Lustre: dagg-OST0002: Connection restored to 7002ccab-a112-d1d5-6fdf-c8c42ad342fc (at 192.168.44.117@o2ib44) Mar 16 03:18:19 arkle2 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:19 john17 kernel: Lustre: dagg-OST0002-osc-ffff881899e5b000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:18:19 john17 kernel: LNetError: 912:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448316931344 length 1048576 too big: 1044508 left, 1044508 allowed Mar 16 03:18:19 john17 kernel: LNetError: 912:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:18:19 john19 kernel: Lustre: 907:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff881d29066f00 x1594448327446320/t0(0) o4->dagg-OST000a-osc-ffff88189a323000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:18:19 john19 kernel: Lustre: dagg-OST000a-osc-ffff88189a323000: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 john26 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff882a53e52700 x1594448281318416/t0(0) o4->dagg-OST0008-osc-ffff882fde832000@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:19 john26 kernel: Lustre: dagg-OST0008-osc-ffff882fde832000: Connection to dagg-OST0008 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88176883c000 Mar 16 03:18:19 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88176883c000 Mar 16 03:18:19 arkle2 kernel: LustreError: 350069:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882ed0b51850 x1594448316853392/t0(0) o4->6b4209e2-620e-030d-e2d1-b9c575162077@192.168.44.117@o2ib44:532/0 lens 608/448 e 0 to 0 dl 1521130742 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:18:19 arkle6 kernel: Lustre: dagg-OST000a: Client d4acd478-18fd-71d3-c177-089c607f8639 (at 192.168.44.119@o2ib44) reconnecting Mar 16 03:18:19 arkle6 kernel: Lustre: dagg-OST000a: Connection restored to e1a18b29-346e-ff08-5424-24003125f711 (at 192.168.44.119@o2ib44) Mar 16 03:18:19 arkle5 kernel: Lustre: dagg-OST0008: Client f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44) reconnecting Mar 16 03:18:19 arkle5 kernel: Lustre: dagg-OST0008: Connection restored to d5de0e2d-4431-41fa-738a-2fa7d0a3c365 (at 192.168.44.126@o2ib44) Mar 16 03:18:19 john18 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff882b86330900 x1594448284367264/t0(0) o4->dagg-OST0000-osc-ffff88015a243000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:19 john18 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:18:19 john18 kernel: Lustre: dagg-OST0000-osc-ffff88015a243000: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177695c200 Mar 16 03:18:19 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ae0ce9600 Mar 16 03:18:19 john26 kernel: Lustre: dagg-OST0008-osc-ffff882fde832000: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:18:19 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177695c200 Mar 16 03:18:19 john18 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:19 arkle2 kernel: LustreError: 350069:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 15 previous similar messages Mar 16 03:18:19 john19 kernel: Lustre: dagg-OST000a-osc-ffff88189a323000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:18:19 john19 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448327510368 length 1048576 too big: 1047928 left, 1047928 allowed Mar 16 03:18:19 john26 kernel: LNetError: 904:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594448281386144 length 1048576 too big: 1045572 left, 1045572 allowed Mar 16 03:18:19 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ae0ce9600 Mar 16 03:18:19 arkle2 kernel: Lustre: dagg-OST0002: Bulk IO write error with 6b4209e2-620e-030d-e2d1-b9c575162077 (at 192.168.44.117@o2ib44), client will retry: rc = -110 Mar 16 03:18:19 arkle2 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:18:19 arkle1 kernel: Lustre: dagg-OST0000: Client 5abb6bc1-b79c-bc3d-60a6-8103bee751c7 (at 192.168.44.118@o2ib44) reconnecting Mar 16 03:18:19 arkle1 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:19 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed87a00 Mar 16 03:18:19 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed87a00 Mar 16 03:18:19 arkle5 kernel: LustreError: 449866:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8817c33d4c50 x1594448281318416/t0(0) o4->f4a3cf53-7286-0d27-5cb4-5f6321872642@192.168.44.126@o2ib44:532/0 lens 608/448 e 0 to 0 dl 1521130742 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:18:19 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882722f94400 Mar 16 03:18:19 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882722f94400 Mar 16 03:18:19 john6 kernel: Lustre: 954:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff8819dca5c200 x1594448323927280/t0(0) o4->dagg-OST0000-osc-ffff881899c73800@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:19 john6 kernel: Lustre: dagg-OST0000-osc-ffff881899c73800: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 arkle1 kernel: Lustre: dagg-OST0000: Connection restored to aad63549-8ffd-bbaa-adc4-c670c37277c9 (at 192.168.44.118@o2ib44) Mar 16 03:18:19 arkle1 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:19 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882efa9aa600 Mar 16 03:18:19 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882efa9aa600 Mar 16 03:18:19 john6 kernel: Lustre: dagg-OST0000-osc-ffff881899c73800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:18:19 john18 kernel: Lustre: dagg-OST0000-osc-ffff88015a243000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:18:19 john18 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:19 john6 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448324004416 length 1048576 too big: 1047972 left, 1047972 allowed Mar 16 03:18:19 arkle5 kernel: Lustre: dagg-OST0008: Bulk IO write error with f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44), client will retry: rc = -110 Mar 16 03:18:19 john16 kernel: Lustre: 943:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff881ab807da00 x1594448321485008/t0(0) o4->dagg-OST0007-osc-ffff882fc87e8000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:19 john16 kernel: Lustre: dagg-OST0007-osc-ffff882fc87e8000: Connection to dagg-OST0007 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 john12 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff8809853abc00 x1594448324464560/t0(0) o4->dagg-OST000a-osc-ffff88015a3ab800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:19 john12 kernel: Lustre: dagg-OST000a-osc-ffff88015a3ab800: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 john12 kernel: Lustre: dagg-OST000a-osc-ffff88015a3ab800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:18:19 john16 kernel: Lustre: dagg-OST0007-osc-ffff882fc87e8000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:18:19 john16 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448321567376 length 1048576 too big: 1048292 left, 1048292 allowed Mar 16 03:18:19 john18 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448284444096 length 1048576 too big: 1048556 left, 1048556 allowed Mar 16 03:18:19 john18 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:18:19 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cac00 Mar 16 03:18:19 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cac00 Mar 16 03:18:19 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c7490aa00 Mar 16 03:18:19 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c7490aa00 Mar 16 03:18:19 arkle1 kernel: LustreError: 435655:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882edd353850 x1594448284367264/t0(0) o4->5abb6bc1-b79c-bc3d-60a6-8103bee751c7@192.168.44.118@o2ib44:532/0 lens 608/448 e 0 to 0 dl 1521130742 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:18:19 arkle1 kernel: Lustre: dagg-OST0000: Bulk IO write error with 5abb6bc1-b79c-bc3d-60a6-8103bee751c7 (at 192.168.44.118@o2ib44), client will retry: rc = -110 Mar 16 03:18:19 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880ffc31d600 Mar 16 03:18:19 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880ffc31d600 Mar 16 03:18:19 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e097a00 Mar 16 03:18:19 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e097a00 Mar 16 03:18:19 john12 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448324534800 length 1048576 too big: 1047560 left, 1047560 allowed Mar 16 03:18:19 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e097a00 Mar 16 03:18:19 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e097a00 Mar 16 03:18:19 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e3de8a600 Mar 16 03:18:19 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e3de8a600 Mar 16 03:18:19 john8 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff88270812b300 x1594448316803760/t0(0) o4->dagg-OST0000-osc-ffff88015a2fe000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:19 john8 kernel: Lustre: dagg-OST0000-osc-ffff88015a2fe000: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808ac2ab400 Mar 16 03:18:19 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808ac2ab400 Mar 16 03:18:19 arkle3 kernel: Lustre: dagg-OST0004: Client faed83b4-3a5e-2a8e-2882-7e99d174dfb0 (at 192.168.44.130@o2ib44) reconnecting Mar 16 03:18:19 arkle3 kernel: Lustre: dagg-OST0004: Connection restored to 2beb6461-856d-b027-2da8-040c1c990221 (at 192.168.44.130@o2ib44) Mar 16 03:18:19 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880c23d30a00 Mar 16 03:18:19 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880c23d30a00 Mar 16 03:18:19 john8 kernel: Lustre: dagg-OST0000-osc-ffff88015a2fe000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:18:19 john8 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448316875328 length 1048576 too big: 1047284 left, 1047284 allowed Mar 16 03:18:19 john8 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:18:19 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8822ef656800 Mar 16 03:18:19 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8822ef656800 Mar 16 03:18:19 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c6c9a8600 Mar 16 03:18:19 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c6c9a8600 Mar 16 03:18:19 arkle3 kernel: LustreError: 235016:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882a77d2f450 x1594699400484576/t0(0) o4->faed83b4-3a5e-2a8e-2882-7e99d174dfb0@192.168.44.130@o2ib44:532/0 lens 608/448 e 0 to 0 dl 1521130742 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:18:19 arkle3 kernel: LustreError: 235016:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 15 previous similar messages Mar 16 03:18:19 john33 kernel: Lustre: 940:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff882e3f367500 x1594699393878992/t0(0) o4->dagg-OST000a-osc-ffff88006bb84800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:19 john33 kernel: Lustre: 940:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:18:19 john33 kernel: Lustre: dagg-OST000a-osc-ffff88006bb84800: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 arkle3 kernel: Lustre: dagg-OST0004: Bulk IO write error with faed83b4-3a5e-2a8e-2882-7e99d174dfb0 (at 192.168.44.130@o2ib44), client will retry: rc = -110 Mar 16 03:18:19 arkle3 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:18:19 john33 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:19 john33 kernel: Lustre: dagg-OST000a-osc-ffff88006bb84800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:18:19 john33 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:19 john32 kernel: Lustre: 928:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff882182fbef00 x1594699396464000/t0(0) o4->dagg-OST000d-osc-ffff8817c5384000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:19 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8822ef654800 Mar 16 03:18:19 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8822ef654800 Mar 16 03:18:19 john32 kernel: Lustre: 928:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:18:19 john32 kernel: Lustre: dagg-OST000d-osc-ffff8817c5384000: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 john32 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:19 john32 kernel: Lustre: dagg-OST000d-osc-ffff8817c5384000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:18:19 john33 kernel: LNetError: 903:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594699393950512 length 1048576 too big: 1046348 left, 1046348 allowed Mar 16 03:18:19 john33 kernel: LNetError: 903:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:18:19 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c6c9a8600 Mar 16 03:18:19 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c6c9a8600 Mar 16 03:18:19 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c65f1fe00 Mar 16 03:18:19 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c65f1fe00 Mar 16 03:18:19 john32 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:19 john9 kernel: Lustre: 935:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff88258cc91b00 x1594448323945264/t0(0) o4->dagg-OST0004-osc-ffff882fdd33b800@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:19 john9 kernel: Lustre: dagg-OST0004-osc-ffff882fdd33b800: Connection to dagg-OST0004 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e3de8a600 Mar 16 03:18:19 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e3de8a600 Mar 16 03:18:19 john29 kernel: Lustre: 936:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff882c3e225a00 x1594699419066352/t0(0) o4->dagg-OST0005-osc-ffff88189afee000@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:19 john32 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594699396533408 length 1048576 too big: 1048540 left, 1048540 allowed Mar 16 03:18:19 john32 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:18:19 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88068c131800 Mar 16 03:18:19 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88068c131800 Mar 16 03:18:19 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882474797e00 Mar 16 03:18:19 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817696a3a00 Mar 16 03:18:19 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817696a3a00 Mar 16 03:18:19 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882474797e00 Mar 16 03:18:19 john29 kernel: Lustre: dagg-OST0005-osc-ffff88189afee000: Connection to dagg-OST0005 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 john29 kernel: Lustre: dagg-OST0005-osc-ffff88189afee000: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:18:19 john29 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594699419120448 length 1048576 too big: 1045736 left, 1045736 allowed Mar 16 03:18:19 john29 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:18:19 john9 kernel: Lustre: dagg-OST0004-osc-ffff882fdd33b800: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:18:19 john9 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448324029520 length 1048576 too big: 1044696 left, 1044696 allowed Mar 16 03:18:19 john9 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:18:19 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c65f1fe00 Mar 16 03:18:19 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c65f1fe00 Mar 16 03:18:19 john13 kernel: Lustre: 935:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff88283c641800 x1594448319199040/t0(0) o4->dagg-OST000d-osc-ffff88015a2c6800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:19 john13 kernel: Lustre: dagg-OST000d-osc-ffff88015a2c6800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 john13 kernel: Lustre: dagg-OST000d-osc-ffff88015a2c6800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:18:19 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817696a3a00 Mar 16 03:18:19 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817696a3a00 Mar 16 03:18:19 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880ffc31d600 Mar 16 03:18:19 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880ffc31d600 Mar 16 03:18:19 john13 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448319269488 length 1048576 too big: 1046108 left, 1046108 allowed Mar 16 03:18:19 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177695f600 Mar 16 03:18:19 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177695f600 Mar 16 03:18:19 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88068c131800 Mar 16 03:18:19 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88068c131800 Mar 16 03:18:19 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817696a3a00 Mar 16 03:18:19 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817696a3a00 Mar 16 03:18:19 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817583a3a00 Mar 16 03:18:19 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817583a3a00 Mar 16 03:18:19 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880583af9000 Mar 16 03:18:19 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880583af9000 Mar 16 03:18:19 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e097a00 Mar 16 03:18:19 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e097a00 Mar 16 03:18:19 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817583a3a00 Mar 16 03:18:19 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817583a3a00 Mar 16 03:18:19 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177695d600 Mar 16 03:18:19 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177695d600 Mar 16 03:18:19 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e097a00 Mar 16 03:18:19 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e097a00 Mar 16 03:18:19 john23 kernel: Lustre: 924:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130655/real 1521130655] req@ffff8814da225d00 x1594448282171968/t0(0) o4->dagg-OST0002-osc-ffff882fdbc22800@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130699 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:19 john23 kernel: Lustre: 924:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:18:19 john23 kernel: Lustre: dagg-OST0002-osc-ffff882fdbc22800: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:19 john23 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:19 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed87a00 Mar 16 03:18:19 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed87a00 Mar 16 03:18:19 john23 kernel: Lustre: dagg-OST0002-osc-ffff882fdbc22800: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:18:19 john23 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:19 john23 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448282253440 length 1048576 too big: 1047668 left, 1047668 allowed Mar 16 03:18:19 john23 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 1 previous similar message Mar 16 03:18:20 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed87a00 Mar 16 03:18:20 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed87a00 Mar 16 03:18:20 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88176883c000 Mar 16 03:18:20 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88176883c000 Mar 16 03:18:20 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8815a47f4a00 Mar 16 03:18:20 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8815a47f4a00 Mar 16 03:18:20 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed87a00 Mar 16 03:18:20 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed87a00 Mar 16 03:18:20 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880ffc31d600 Mar 16 03:18:20 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880bafeff400 Mar 16 03:18:20 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880583af9000 Mar 16 03:18:20 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894a8800 Mar 16 03:18:20 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880ffc31d600 Mar 16 03:18:20 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880bafeff400 Mar 16 03:18:20 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880583af9000 Mar 16 03:18:20 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894a8800 Mar 16 03:18:20 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88176883c000 Mar 16 03:18:20 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88176883c000 Mar 16 03:18:20 john31 kernel: Lustre: 936:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130656/real 1521130656] req@ffff882758f22a00 x1594699394168800/t0(0) o4->dagg-OST0003-osc-ffff882fde9fd000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130700 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:20 john31 kernel: Lustre: dagg-OST0003-osc-ffff882fde9fd000: Connection to dagg-OST0003 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:20 john31 kernel: Lustre: dagg-OST0003-osc-ffff882fde9fd000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:18:20 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cba00 Mar 16 03:18:20 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cba00 Mar 16 03:18:20 john31 kernel: LNetError: 895:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594699394239776 length 1048576 too big: 1048180 left, 1048180 allowed Mar 16 03:18:20 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8827c479cc00 Mar 16 03:18:20 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8827c479cc00 Mar 16 03:18:20 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88176883c000 Mar 16 03:18:20 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cc600 Mar 16 03:18:20 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cc600 Mar 16 03:18:20 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88176883c000 Mar 16 03:18:20 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed87a00 Mar 16 03:18:20 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e093000 Mar 16 03:18:20 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e093000 Mar 16 03:18:20 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed87a00 Mar 16 03:18:20 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ca5823800 Mar 16 03:18:20 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ca5823800 Mar 16 03:18:20 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c7490aa00 Mar 16 03:18:20 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c7490aa00 Mar 16 03:18:20 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c7490aa00 Mar 16 03:18:20 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8815a47f4a00 Mar 16 03:18:20 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8815a47f4a00 Mar 16 03:18:20 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8827c479cc00 Mar 16 03:18:20 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8827c479cc00 Mar 16 03:18:20 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c7490aa00 Mar 16 03:18:20 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88068c136e00 Mar 16 03:18:20 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88068c136e00 Mar 16 03:18:20 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a426400 Mar 16 03:18:20 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a426400 Mar 16 03:18:20 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881852ebe000 Mar 16 03:18:20 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881852ebe000 Mar 16 03:18:20 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c7490aa00 Mar 16 03:18:20 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c7490aa00 Mar 16 03:18:20 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881fc0c30000 Mar 16 03:18:20 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881fc0c30000 Mar 16 03:18:21 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880ffc31d600 Mar 16 03:18:21 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880bafeff400 Mar 16 03:18:21 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880ffc31d600 Mar 16 03:18:21 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880bafeff400 Mar 16 03:18:21 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8828607d8400 Mar 16 03:18:21 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8828607d8400 Mar 16 03:18:21 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8824162c5c00 Mar 16 03:18:21 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8824162c5c00 Mar 16 03:18:21 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88068c133e00 Mar 16 03:18:21 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88068c133e00 Mar 16 03:18:21 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88261401c800 Mar 16 03:18:21 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88261401c800 Mar 16 03:18:21 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ca5823800 Mar 16 03:18:21 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ca5823800 Mar 16 03:18:21 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e093000 Mar 16 03:18:21 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e093000 Mar 16 03:18:21 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a425400 Mar 16 03:18:21 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a425400 Mar 16 03:18:21 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808ac2a9200 Mar 16 03:18:21 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808ac2a9200 Mar 16 03:18:21 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a420e00 Mar 16 03:18:21 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a420e00 Mar 16 03:18:22 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac953000 Mar 16 03:18:22 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac953000 Mar 16 03:18:22 john10 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130657/real 1521130657] req@ffff881757b33300 x1594448319886272/t0(0) o4->dagg-OST000f-osc-ffff882fd4b83800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130701 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:22 john10 kernel: Lustre: dagg-OST0001-osc-ffff882fd4b83800: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:22 john10 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:22 john10 kernel: Lustre: dagg-OST0001-osc-ffff882fd4b83800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:18:22 john10 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:22 john10 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448319957872 length 1048576 too big: 1046492 left, 1046492 allowed Mar 16 03:18:22 john10 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:18:22 john10 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:18:22 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808ac2a9200 Mar 16 03:18:22 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808ac2a9200 Mar 16 03:18:22 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cc600 Mar 16 03:18:22 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cc600 Mar 16 03:18:22 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817583a3e00 Mar 16 03:18:22 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817583a3e00 Mar 16 03:18:22 john14 kernel: Lustre: 929:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130658/real 1521130658] req@ffff880beba30f00 x1594448317088528/t0(0) o4->dagg-OST000b-osc-ffff88189a368800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130702 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:22 john14 kernel: Lustre: 929:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:18:22 john14 kernel: Lustre: dagg-OST000b-osc-ffff88189a368800: Connection to dagg-OST000b (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:22 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e093000 Mar 16 03:18:22 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e093000 Mar 16 03:18:22 john14 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:22 john14 kernel: Lustre: dagg-OST000b-osc-ffff88189a368800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:18:22 john14 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:22 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177695f400 Mar 16 03:18:22 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177695f400 Mar 16 03:18:22 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e4405b600 Mar 16 03:18:22 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e4405b600 Mar 16 03:18:22 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ca5823800 Mar 16 03:18:22 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ca5823800 Mar 16 03:18:22 john13 kernel: Lustre: 931:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130658/real 1521130658] req@ffff8801a13e6300 x1594448319201264/t0(0) o4->dagg-OST0000-osc-ffff88015a2c6800@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130702 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:22 john13 kernel: Lustre: 931:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:18:22 john13 kernel: Lustre: dagg-OST0000-osc-ffff88015a2c6800: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:22 john13 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:18:22 john13 kernel: Lustre: dagg-OST0000-osc-ffff88015a2c6800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:18:22 john13 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:18:22 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cc600 Mar 16 03:18:22 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cc600 Mar 16 03:18:22 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881852eb9600 Mar 16 03:18:22 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881852eb9600 Mar 16 03:18:23 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e4405b600 Mar 16 03:18:23 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e4405b600 Mar 16 03:18:23 john6 kernel: Lustre: 953:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130659/real 1521130659] req@ffff8819dca5fb00 x1594448323934928/t0(0) o4->dagg-OST0009-osc-ffff881899c73800@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130703 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:23 john6 kernel: Lustre: 953:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:18:23 john6 kernel: Lustre: dagg-OST0009-osc-ffff881899c73800: Connection to dagg-OST0009 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:23 john6 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:18:23 john6 kernel: Lustre: dagg-OST0009-osc-ffff881899c73800: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:18:23 john6 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:18:23 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed86400 Mar 16 03:18:23 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed86400 Mar 16 03:18:23 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ca5823800 Mar 16 03:18:23 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ca5823800 Mar 16 03:18:24 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a427c00 Mar 16 03:18:24 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a427c00 Mar 16 03:18:24 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac953000 Mar 16 03:18:24 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac953000 Mar 16 03:18:24 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88068c133e00 Mar 16 03:18:24 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88068c133e00 Mar 16 03:18:24 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881efe3b6c00 Mar 16 03:18:24 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881efe3b6c00 Mar 16 03:18:25 john10 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130660/real 1521130660] req@ffff882288a98300 x1594448319890080/t0(0) o4->dagg-OST0008-osc-ffff882fd4b83800@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130704 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:25 john10 kernel: Lustre: dagg-OST0008-osc-ffff882fd4b83800: Connection to dagg-OST0008 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:25 john10 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:25 john10 kernel: Lustre: dagg-OST0008-osc-ffff882fd4b83800: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:18:25 john10 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:18:25 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a426800 Mar 16 03:18:25 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a426800 Mar 16 03:18:25 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8824162c3200 Mar 16 03:18:25 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8824162c3200 Mar 16 03:18:26 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881cc1d1d400 Mar 16 03:18:26 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881cc1d1d400 Mar 16 03:18:26 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ca5820800 Mar 16 03:18:26 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ca5820800 Mar 16 03:18:26 john9 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130662/real 1521130662] req@ffff8822bf6a1b00 x1594448323976576/t0(0) o4->dagg-OST000a-osc-ffff882fdd33b800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130706 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:26 john9 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:18:26 john9 kernel: Lustre: dagg-OST000a-osc-ffff882fdd33b800: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:26 john9 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:26 john9 kernel: Lustre: dagg-OST000a-osc-ffff882fdd33b800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:18:26 john9 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:26 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ca5820800 Mar 16 03:18:26 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ca5820800 Mar 16 03:18:26 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ca5820800 Mar 16 03:18:26 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ca5820800 Mar 16 03:18:27 john17 kernel: Lustre: 956:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130663/real 1521130663] req@ffff8824f3bcb000 x1594448316863856/t0(0) o4->dagg-OST0003-osc-ffff881899e5b000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130707 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:27 john17 kernel: Lustre: 956:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:18:27 john17 kernel: Lustre: dagg-OST0003-osc-ffff881899e5b000: Connection to dagg-OST0003 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:27 john17 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:27 arkle2 kernel: Lustre: dagg-OST0003: Client 6b4209e2-620e-030d-e2d1-b9c575162077 (at 192.168.44.117@o2ib44) reconnecting Mar 16 03:18:27 arkle2 kernel: Lustre: Skipped 12 previous similar messages Mar 16 03:18:27 arkle2 kernel: Lustre: dagg-OST0003: Connection restored to 7002ccab-a112-d1d5-6fdf-c8c42ad342fc (at 192.168.44.117@o2ib44) Mar 16 03:18:27 arkle2 kernel: Lustre: Skipped 12 previous similar messages Mar 16 03:18:27 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881efe3b7800 Mar 16 03:18:27 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881efe3b7800 Mar 16 03:18:27 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8824162c3200 Mar 16 03:18:27 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8824162c3200 Mar 16 03:18:27 john17 kernel: Lustre: dagg-OST0003-osc-ffff881899e5b000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:18:27 john17 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:27 john22 kernel: Lustre: 931:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130663/real 1521130663] req@ffff880c93ec4e00 x1594448280693824/t0(0) o4->dagg-OST0001-osc-ffff88189a897000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130707 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:27 john22 kernel: Lustre: 931:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:18:27 john22 kernel: Lustre: dagg-OST0001-osc-ffff88189a897000: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:27 john22 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:27 arkle1 kernel: Lustre: dagg-OST0001: Client 53090394-a3ef-6106-82db-77d8ecba9e98 (at 192.168.44.122@o2ib44) reconnecting Mar 16 03:18:27 arkle1 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:18:27 arkle1 kernel: Lustre: dagg-OST0001: Connection restored to 2bbb92b8-678a-04cd-06aa-2b4a72b2566f (at 192.168.44.122@o2ib44) Mar 16 03:18:27 arkle1 kernel: Lustre: Skipped 12 previous similar messages Mar 16 03:18:28 john22 kernel: Lustre: dagg-OST0001-osc-ffff88189a897000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:18:28 john22 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:28 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18cc600 Mar 16 03:18:28 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18cc600 Mar 16 03:18:28 john28 kernel: Lustre: 936:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130664/real 1521130664] req@ffff8819a6e74200 x1594448282279632/t0(0) o4->dagg-OST0005-osc-ffff8817d9cf4000@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130708 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:28 john28 kernel: Lustre: 936:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:18:28 john28 kernel: Lustre: dagg-OST0005-osc-ffff8817d9cf4000: Connection to dagg-OST0005 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:28 john28 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:18:28 john28 kernel: Lustre: dagg-OST0005-osc-ffff8817d9cf4000: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:18:28 john28 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:18:28 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881cc1d1f400 Mar 16 03:18:28 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881cc1d1f400 Mar 16 03:18:30 john16 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130666/real 1521130666] req@ffff882adee1a400 x1594448321516688/t0(0) o4->dagg-OST0009-osc-ffff882fc87e8000@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130710 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:30 john16 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:18:30 john16 kernel: Lustre: dagg-OST0009-osc-ffff882fc87e8000: Connection to dagg-OST0009 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:30 john16 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:30 john16 kernel: Lustre: dagg-OST0009-osc-ffff882fc87e8000: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:18:30 john16 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:30 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed81c00 Mar 16 03:18:30 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed81c00 Mar 16 03:18:31 john19 kernel: Lustre: 908:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130667/real 1521130667] req@ffff881d29064b00 x1594448327445664/t0(0) o4->dagg-OST000d-osc-ffff88189a323000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130711 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:31 john19 kernel: Lustre: dagg-OST000d-osc-ffff88189a323000: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:31 john19 kernel: Lustre: dagg-OST000d-osc-ffff88189a323000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:18:31 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88068c137200 Mar 16 03:18:31 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88068c137200 Mar 16 03:18:39 john26 kernel: Lustre: 942:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130675/real 1521130675] req@ffff881dfc3ff800 x1594448281339792/t0(0) o4->dagg-OST000e-osc-ffff882fde832000@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130719 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:39 john26 kernel: Lustre: 942:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:18:39 john26 kernel: Lustre: dagg-OST000e-osc-ffff882fde832000: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:39 john26 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:39 arkle8 kernel: Lustre: dagg-OST000e: Client f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44) reconnecting Mar 16 03:18:39 arkle8 kernel: Lustre: Skipped 12 previous similar messages Mar 16 03:18:39 arkle8 kernel: Lustre: dagg-OST000e: Connection restored to d5de0e2d-4431-41fa-738a-2fa7d0a3c365 (at 192.168.44.126@o2ib44) Mar 16 03:18:39 arkle8 kernel: Lustre: Skipped 6 previous similar messages Mar 16 03:18:39 john26 kernel: Lustre: dagg-OST000e-osc-ffff882fde832000: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:18:39 john26 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:39 john26 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448281408192 length 1048576 too big: 1047960 left, 1047960 allowed Mar 16 03:18:39 john26 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 2 previous similar messages Mar 16 03:18:39 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88095c0e7a00 Mar 16 03:18:39 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88095c0e7a00 Mar 16 03:18:39 arkle8 kernel: LustreError: 309857:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8805a52e8c50 x1594448281339792/t0(0) o4->f4a3cf53-7286-0d27-5cb4-5f6321872642@192.168.44.126@o2ib44:552/0 lens 608/448 e 0 to 0 dl 1521130762 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:18:39 arkle8 kernel: LustreError: 309857:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 13 previous similar messages Mar 16 03:18:39 arkle8 kernel: Lustre: dagg-OST000e: Bulk IO write error with f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44), client will retry: rc = -110 Mar 16 03:18:39 arkle8 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:18:39 john11 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130675/real 1521130675] req@ffff8826ea14ad00 x1594448320168128/t0(0) o4->dagg-OST000a-osc-ffff882fd571f000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130719 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:39 john11 kernel: Lustre: dagg-OST000a-osc-ffff882fd571f000: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:39 arkle6 kernel: Lustre: dagg-OST000a: Client 7c2b1b78-121a-b208-fc8a-57295d42d7f7 (at 192.168.44.111@o2ib44) reconnecting Mar 16 03:18:39 arkle6 kernel: Lustre: Skipped 12 previous similar messages Mar 16 03:18:39 arkle6 kernel: Lustre: dagg-OST000a: Connection restored to 9e90df21-9b59-537c-f486-0efeb807faad (at 192.168.44.111@o2ib44) Mar 16 03:18:39 arkle6 kernel: Lustre: Skipped 12 previous similar messages Mar 16 03:18:39 john11 kernel: Lustre: dagg-OST000a-osc-ffff882fd571f000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:18:39 john11 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448320267360 length 1048576 too big: 1047284 left, 1047284 allowed Mar 16 03:18:39 john11 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:18:39 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177695d800 Mar 16 03:18:39 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177695d800 Mar 16 03:18:39 arkle6 kernel: LustreError: 330393:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88101f942c50 x1594448320168128/t0(0) o4->7c2b1b78-121a-b208-fc8a-57295d42d7f7@192.168.44.111@o2ib44:552/0 lens 608/448 e 0 to 0 dl 1521130762 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:18:39 arkle6 kernel: LustreError: 330393:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 15 previous similar messages Mar 16 03:18:39 arkle6 kernel: Lustre: dagg-OST000a: Bulk IO write error with 7c2b1b78-121a-b208-fc8a-57295d42d7f7 (at 192.168.44.111@o2ib44), client will retry: rc = -110 Mar 16 03:18:39 arkle6 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:18:39 john25 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130675/real 1521130675] req@ffff8824baa20f00 x1594448280895840/t0(0) o4->dagg-OST000e-osc-ffff882fd7db5800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130719 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:39 john25 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:18:39 john25 kernel: Lustre: dagg-OST000e-osc-ffff882fd7db5800: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:39 john25 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:18:39 john25 kernel: Lustre: dagg-OST000e-osc-ffff882fd7db5800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:18:39 john25 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:18:39 john25 kernel: LNetError: 903:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448280964368 length 1048576 too big: 1046340 left, 1046340 allowed Mar 16 03:18:39 john25 kernel: LNetError: 903:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:18:39 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8827c2ba8a00 Mar 16 03:18:39 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8827c2ba8a00 Mar 16 03:18:40 john12 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130676/real 1521130676] req@ffff881f017c1800 x1594448324471616/t0(0) o4->dagg-OST000e-osc-ffff88015a3ab800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130720 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:40 john12 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:18:40 john12 kernel: Lustre: dagg-OST000e-osc-ffff88015a3ab800: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:40 john12 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:40 john12 kernel: Lustre: dagg-OST000e-osc-ffff88015a3ab800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:18:40 john12 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:40 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882798482e00 Mar 16 03:18:40 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882798482e00 Mar 16 03:18:40 john31 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130676/real 1521130676] req@ffff881c56f45100 x1594699394190000/t0(0) o4->dagg-OST000e-osc-ffff882fde9fd000@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130720 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:40 john31 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:18:40 john31 kernel: Lustre: dagg-OST000e-osc-ffff882fde9fd000: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:40 john31 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:40 john31 kernel: Lustre: dagg-OST000e-osc-ffff882fde9fd000: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:18:40 john31 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:40 john31 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594699394259008 length 1048576 too big: 1048208 left, 1048208 allowed Mar 16 03:18:40 john31 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 2 previous similar messages Mar 16 03:18:40 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882798482e00 Mar 16 03:18:40 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882798482e00 Mar 16 03:18:41 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88095c0e3800 Mar 16 03:18:41 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8812019ac400 Mar 16 03:18:41 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88095c0e3800 Mar 16 03:18:41 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8812019ac400 Mar 16 03:18:41 john15 kernel: Lustre: 919:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130677/real 1521130677] req@ffff880e86ebb300 x1594448323943024/t0(0) o4->dagg-OST000e-osc-ffff882fde9f4800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130721 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:18:41 john15 kernel: Lustre: 919:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:18:41 john15 kernel: Lustre: dagg-OST000e-osc-ffff882fde9f4800: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:18:41 john15 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:41 john15 kernel: Lustre: dagg-OST000e-osc-ffff882fde9f4800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:18:41 john15 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:18:41 john15 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448324011200 length 1048576 too big: 1046392 left, 1046392 allowed Mar 16 03:18:41 john15 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 2 previous similar messages Mar 16 03:18:42 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e6bc5fc00 Mar 16 03:18:42 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e6bc5fc00 Mar 16 03:18:42 arkle3 kernel: Lustre: dagg-OST0004: Client 7c2b1b78-121a-b208-fc8a-57295d42d7f7 (at 192.168.44.111@o2ib44) reconnecting Mar 16 03:18:42 arkle3 kernel: Lustre: Skipped 11 previous similar messages Mar 16 03:18:42 arkle3 kernel: Lustre: dagg-OST0004: Connection restored to 9e90df21-9b59-537c-f486-0efeb807faad (at 192.168.44.111@o2ib44) Mar 16 03:18:42 arkle3 kernel: Lustre: Skipped 11 previous similar messages Mar 16 03:18:42 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817583a6400 Mar 16 03:18:42 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817583a6400 Mar 16 03:19:02 john25 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130698/real 1521130698] req@ffff881574af9e00 x1594448280872016/t0(0) o4->dagg-OST0006-osc-ffff882fd7db5800@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130742 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:02 john25 kernel: Lustre: dagg-OST0006-osc-ffff882fd7db5800: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:02 arkle4 kernel: Lustre: dagg-OST0006: Client 40d9b451-fab3-be40-89fb-0c9b04087671 (at 192.168.44.125@o2ib44) reconnecting Mar 16 03:19:02 arkle4 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:19:02 arkle4 kernel: Lustre: dagg-OST0006: Connection restored to edb2edda-7927-928c-4497-0706e1cfa6ce (at 192.168.44.125@o2ib44) Mar 16 03:19:02 arkle4 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:19:02 john14 kernel: Lustre: 937:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130698/real 1521130698] req@ffff882dddbaef00 x1594448317079152/t0(0) o4->dagg-OST0006-osc-ffff88189a368800@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130742 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:19:02 john14 kernel: Lustre: dagg-OST0006-osc-ffff88189a368800: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:02 john25 kernel: Lustre: dagg-OST0006-osc-ffff882fd7db5800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:19:02 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e281e7200 Mar 16 03:19:02 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e281e7200 Mar 16 03:19:02 arkle4 kernel: LustreError: 302876:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882ddb73ac50 x1594448280872016/t0(0) o4->40d9b451-fab3-be40-89fb-0c9b04087671@192.168.44.125@o2ib44:575/0 lens 608/448 e 0 to 0 dl 1521130785 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:19:02 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e091c00 Mar 16 03:19:02 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e091c00 Mar 16 03:19:02 john14 kernel: Lustre: dagg-OST0006-osc-ffff88189a368800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:19:02 john30 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130698/real 1521130698] req@ffff882e4338a400 x1594699400481984/t0(0) o4->dagg-OST000c-osc-ffff88189ac92000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130742 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:02 john30 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:02 john30 kernel: Lustre: dagg-OST000c-osc-ffff88189ac92000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:02 john14 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448317195552 length 1048576 too big: 1045352 left, 1045352 allowed Mar 16 03:19:02 john14 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:02 john30 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:02 arkle7 kernel: Lustre: dagg-OST000c: Client faed83b4-3a5e-2a8e-2882-7e99d174dfb0 (at 192.168.44.130@o2ib44) reconnecting Mar 16 03:19:02 arkle7 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:02 arkle7 kernel: Lustre: dagg-OST000c: Connection restored to 2beb6461-856d-b027-2da8-040c1c990221 (at 192.168.44.130@o2ib44) Mar 16 03:19:02 arkle7 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:02 john28 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130698/real 1521130698] req@ffff881aaab92d00 x1594448282246752/t0(0) o4->dagg-OST0006-osc-ffff8817d9cf4000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130742 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:02 john28 kernel: Lustre: dagg-OST0006-osc-ffff8817d9cf4000: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:02 arkle4 kernel: Lustre: dagg-OST0006: Bulk IO write error with 23017e25-beb4-2d0c-1e7b-9a1bd7cbb415 (at 192.168.44.114@o2ib44), client will retry: rc = -110 Mar 16 03:19:02 arkle4 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:02 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e091c00 Mar 16 03:19:02 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e091c00 Mar 16 03:19:02 arkle4 kernel: LustreError: 302876:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 17 previous similar messages Mar 16 03:19:02 john28 kernel: Lustre: dagg-OST0006-osc-ffff8817d9cf4000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:19:02 john30 kernel: Lustre: dagg-OST000c-osc-ffff88189ac92000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:02 john30 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:02 john30 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594699400579072 length 1048576 too big: 1048516 left, 1048516 allowed Mar 16 03:19:02 john30 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:02 john28 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448282371584 length 1048576 too big: 1045344 left, 1045344 allowed Mar 16 03:19:02 john28 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:02 john27 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130698/real 1521130698] req@ffff881b4e2d3600 x1594448285369712/t0(0) o4->dagg-OST000f-osc-ffff88015a1e4800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130742 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:02 john27 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:02 john27 kernel: Lustre: dagg-OST000f-osc-ffff88015a1e4800: Connection to dagg-OST000f (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:02 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8815f1876800 Mar 16 03:19:02 john27 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:02 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e281e7200 Mar 16 03:19:02 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e281e7200 Mar 16 03:19:02 arkle8 kernel: Lustre: dagg-OST000f: Connection restored to c8b7715d-04bb-a76a-3db9-c45127314e23 (at 192.168.44.127@o2ib44) Mar 16 03:19:02 arkle8 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:19:02 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8815f1876800 Mar 16 03:19:02 arkle7 kernel: LustreError: 34887:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882ed22b3c50 x1594699400481984/t0(0) o4->faed83b4-3a5e-2a8e-2882-7e99d174dfb0@192.168.44.130@o2ib44:575/0 lens 608/448 e 0 to 0 dl 1521130785 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:19:02 arkle7 kernel: LustreError: 34887:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 15 previous similar messages Mar 16 03:19:02 john20 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130698/real 1521130698] req@ffff8828a2b52100 x1594448327525968/t0(0) o4->dagg-OST000d-osc-ffff8817da6ad800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130742 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:02 john27 kernel: Lustre: dagg-OST000f-osc-ffff88015a1e4800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:19:02 john27 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:02 john20 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:02 john20 kernel: Lustre: dagg-OST000d-osc-ffff8817da6ad800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:02 john20 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:02 john20 kernel: Lustre: dagg-OST000d-osc-ffff8817da6ad800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:02 arkle7 kernel: Lustre: dagg-OST000c: Bulk IO write error with faed83b4-3a5e-2a8e-2882-7e99d174dfb0 (at 192.168.44.130@o2ib44), client will retry: rc = -110 Mar 16 03:19:02 arkle7 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:02 john20 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:02 john27 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448285483184 length 1048576 too big: 1046096 left, 1046096 allowed Mar 16 03:19:02 john27 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:02 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8812019ad200 Mar 16 03:19:02 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8812019ad200 Mar 16 03:19:02 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88068c131000 Mar 16 03:19:02 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88068c131000 Mar 16 03:19:02 john20 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448327669648 length 1048576 too big: 1045364 left, 1045364 allowed Mar 16 03:19:02 john20 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:02 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880710ec3000 Mar 16 03:19:02 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880710ec3000 Mar 16 03:19:02 john15 kernel: Lustre: 949:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130698/real 1521130698] req@ffff88207ef11200 x1594448323936064/t0(0) o4->dagg-OST000d-osc-ffff882fde9f4800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130742 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:02 john15 kernel: Lustre: dagg-OST000d-osc-ffff882fde9f4800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:02 john15 kernel: Lustre: dagg-OST000d-osc-ffff882fde9f4800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:02 john7 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130698/real 1521130698] req@ffff8817a06ff500 x1594448319400640/t0(0) o4->dagg-OST0006-osc-ffff882fda4b4000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130742 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:02 john7 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:02 john7 kernel: Lustre: dagg-OST0006-osc-ffff882fda4b4000: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:02 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880710ec3000 Mar 16 03:19:02 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880710ec3000 Mar 16 03:19:02 john7 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:02 john24 kernel: Lustre: 951:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130698/real 1521130698] req@ffff882c67f3f500 x1594448284013248/t0(0) o4->dagg-OST000c-osc-ffff882fcb645000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130742 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:02 john24 kernel: Lustre: 951:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:02 john24 kernel: Lustre: dagg-OST000c-osc-ffff882fcb645000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:02 john24 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:03 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e21a6e000 Mar 16 03:19:03 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e21a6e000 Mar 16 03:19:03 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88068c131000 Mar 16 03:19:03 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88068c131000 Mar 16 03:19:03 john22 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130698/real 1521130698] req@ffff8823b403b900 x1594448280685024/t0(0) o4->dagg-OST000c-osc-ffff88189a897000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130742 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:03 john22 kernel: Lustre: dagg-OST000c-osc-ffff88189a897000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 john7 kernel: Lustre: dagg-OST0006-osc-ffff882fda4b4000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:19:03 john7 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:03 john24 kernel: Lustre: dagg-OST000c-osc-ffff882fcb645000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:03 john24 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:03 john22 kernel: Lustre: dagg-OST000c-osc-ffff88189a897000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:03 john7 kernel: LNetError: 914:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448319503904 length 1048576 too big: 1045464 left, 1045464 allowed Mar 16 03:19:03 john7 kernel: LNetError: 914:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:03 john24 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448284128400 length 1048576 too big: 1046748 left, 1046748 allowed Mar 16 03:19:03 john24 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:03 john22 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448280784496 length 1048576 too big: 1048436 left, 1048436 allowed Mar 16 03:19:03 john22 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:03 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e21a6e000 Mar 16 03:19:03 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88068c131000 Mar 16 03:19:03 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e21a6e000 Mar 16 03:19:03 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88068c131000 Mar 16 03:19:03 john21 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130698/real 1521130698] req@ffff881e6e212d00 x1594448332911728/t0(0) o4->dagg-OST000f-osc-ffff882fc5588800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130742 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:03 john21 kernel: Lustre: dagg-OST000c-osc-ffff882fc5588800: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 john21 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:03 john21 kernel: Lustre: dagg-OST000c-osc-ffff882fc5588800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:03 john21 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:03 john21 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448333010048 length 1048576 too big: 1044712 left, 1044712 allowed Mar 16 03:19:03 john21 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:03 john21 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:03 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88158d996a00 Mar 16 03:19:03 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88158d996a00 Mar 16 03:19:03 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88074211ac00 Mar 16 03:19:03 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88074211ac00 Mar 16 03:19:03 john17 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff880e00b31500 x1594448316853392/t0(0) o4->dagg-OST0002-osc-ffff881899e5b000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:03 john17 kernel: Lustre: dagg-OST0002-osc-ffff881899e5b000: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 arkle2 kernel: Lustre: dagg-OST0002: Client 6b4209e2-620e-030d-e2d1-b9c575162077 (at 192.168.44.117@o2ib44) reconnecting Mar 16 03:19:03 arkle2 kernel: Lustre: dagg-OST0002: Connection restored to 7002ccab-a112-d1d5-6fdf-c8c42ad342fc (at 192.168.44.117@o2ib44) Mar 16 03:19:03 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ae0cee600 Mar 16 03:19:03 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882b144aea00 Mar 16 03:19:03 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ae0cee600 Mar 16 03:19:03 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882b144aea00 Mar 16 03:19:03 john17 kernel: Lustre: dagg-OST0002-osc-ffff881899e5b000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:19:03 john17 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448316972128 length 1048576 too big: 1044508 left, 1044508 allowed Mar 16 03:19:03 john17 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:03 john26 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff882a53e52700 x1594448281318416/t0(0) o4->dagg-OST0008-osc-ffff882fde832000@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:03 john26 kernel: Lustre: dagg-OST0008-osc-ffff882fde832000: Connection to dagg-OST0008 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881768838200 Mar 16 03:19:03 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881768838200 Mar 16 03:19:03 arkle2 kernel: LustreError: 350069:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882ee03f1850 x1594448316853392/t0(0) o4->6b4209e2-620e-030d-e2d1-b9c575162077@192.168.44.117@o2ib44:576/0 lens 608/448 e 0 to 0 dl 1521130786 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:19:03 arkle5 kernel: Lustre: dagg-OST0008: Client f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44) reconnecting Mar 16 03:19:03 arkle5 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:03 arkle5 kernel: Lustre: dagg-OST0008: Connection restored to d5de0e2d-4431-41fa-738a-2fa7d0a3c365 (at 192.168.44.126@o2ib44) Mar 16 03:19:03 john19 kernel: Lustre: 907:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff881d29066f00 x1594448327446320/t0(0) o4->dagg-OST000a-osc-ffff88189a323000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:19:03 john19 kernel: Lustre: dagg-OST000a-osc-ffff88189a323000: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 john19 kernel: Lustre: dagg-OST000a-osc-ffff88189a323000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:19:03 john19 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448327537024 length 1048576 too big: 1047928 left, 1047928 allowed Mar 16 03:19:03 john19 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:03 john19 kernel: Lustre: 907:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:19:03 arkle5 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:03 arkle2 kernel: LustreError: 350069:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 15 previous similar messages Mar 16 03:19:03 john12 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff8809853abc00 x1594448324464560/t0(0) o4->dagg-OST000a-osc-ffff88015a3ab800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:03 john12 kernel: Lustre: dagg-OST000a-osc-ffff88015a3ab800: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 john12 kernel: Lustre: dagg-OST000a-osc-ffff88015a3ab800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:19:03 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ae0cee600 Mar 16 03:19:03 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ae0cee600 Mar 16 03:19:03 john26 kernel: Lustre: dagg-OST0008-osc-ffff882fde832000: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:19:03 arkle2 kernel: Lustre: dagg-OST0002: Bulk IO write error with 6b4209e2-620e-030d-e2d1-b9c575162077 (at 192.168.44.117@o2ib44), client will retry: rc = -110 Mar 16 03:19:03 arkle2 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:03 john18 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff882b86330900 x1594448284367264/t0(0) o4->dagg-OST0000-osc-ffff88015a243000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:03 john18 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:03 john18 kernel: Lustre: dagg-OST0000-osc-ffff88015a243000: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed84a00 Mar 16 03:19:03 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed84a00 Mar 16 03:19:03 arkle5 kernel: LustreError: 449866:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8809084d3c50 x1594448281318416/t0(0) o4->f4a3cf53-7286-0d27-5cb4-5f6321872642@192.168.44.126@o2ib44:576/0 lens 608/448 e 0 to 0 dl 1521130786 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:19:03 john12 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448324561904 length 1048576 too big: 1047560 left, 1047560 allowed Mar 16 03:19:03 john12 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:03 john16 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff881e67300f00 x1594448321487984/t0(0) o4->dagg-OST0007-osc-ffff882fc87e8000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:19:03 john16 kernel: Lustre: dagg-OST0007-osc-ffff882fc87e8000: Connection to dagg-OST0007 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 john18 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:03 john16 kernel: Lustre: dagg-OST0007-osc-ffff882fc87e8000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:19:03 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e6bc5ee00 Mar 16 03:19:03 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e6bc5ee00 Mar 16 03:19:03 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ca5823a00 Mar 16 03:19:03 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ca5823a00 Mar 16 03:19:03 arkle5 kernel: LustreError: 449866:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 15 previous similar messages Mar 16 03:19:03 arkle1 kernel: Lustre: dagg-OST0000: Client 5abb6bc1-b79c-bc3d-60a6-8103bee751c7 (at 192.168.44.118@o2ib44) reconnecting Mar 16 03:19:03 arkle1 kernel: Lustre: dagg-OST0000: Connection restored to aad63549-8ffd-bbaa-adc4-c670c37277c9 (at 192.168.44.118@o2ib44) Mar 16 03:19:03 arkle5 kernel: Lustre: dagg-OST0008: Bulk IO write error with f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44), client will retry: rc = -110 Mar 16 03:19:03 arkle5 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:03 john16 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448321608272 length 1048576 too big: 1048292 left, 1048292 allowed Mar 16 03:19:03 john16 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:03 john18 kernel: Lustre: dagg-OST0000-osc-ffff88015a243000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:19:03 john18 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:03 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b89600 Mar 16 03:19:03 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e097800 Mar 16 03:19:03 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b89600 Mar 16 03:19:03 john18 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448284488080 length 1048576 too big: 1048556 left, 1048556 allowed Mar 16 03:19:03 john18 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:03 john8 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff88270812b300 x1594448316803760/t0(0) o4->dagg-OST0000-osc-ffff88015a2fe000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:03 john8 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:03 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e097800 Mar 16 03:19:03 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18c9e00 Mar 16 03:19:03 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18c9e00 Mar 16 03:19:03 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac953600 Mar 16 03:19:03 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac953600 Mar 16 03:19:03 arkle1 kernel: LustreError: 435655:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882ee0d3c050 x1594448284367264/t0(0) o4->5abb6bc1-b79c-bc3d-60a6-8103bee751c7@192.168.44.118@o2ib44:576/0 lens 608/448 e 0 to 0 dl 1521130786 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:19:03 arkle1 kernel: LustreError: 435655:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 14 previous similar messages Mar 16 03:19:03 arkle1 kernel: Lustre: dagg-OST0000: Bulk IO write error with 5abb6bc1-b79c-bc3d-60a6-8103bee751c7 (at 192.168.44.118@o2ib44), client will retry: rc = -110 Mar 16 03:19:03 arkle1 kernel: Lustre: Skipped 14 previous similar messages Mar 16 03:19:03 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88279a5d2600 Mar 16 03:19:03 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88279a5d2600 Mar 16 03:19:03 john8 kernel: Lustre: dagg-OST0000-osc-ffff88015a2fe000: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 john8 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:03 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e21a6e000 Mar 16 03:19:03 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e21a6e000 Mar 16 03:19:03 john8 kernel: Lustre: dagg-OST0000-osc-ffff88015a2fe000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:19:03 john8 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:03 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18c9e00 Mar 16 03:19:03 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18c9e00 Mar 16 03:19:03 john8 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448316902032 length 1048576 too big: 1047284 left, 1047284 allowed Mar 16 03:19:03 john8 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:03 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a426e00 Mar 16 03:19:03 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a426e00 Mar 16 03:19:03 john33 kernel: Lustre: 940:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff882e3f367500 x1594699393878992/t0(0) o4->dagg-OST000a-osc-ffff88006bb84800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:03 john33 kernel: Lustre: 940:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:03 john33 kernel: Lustre: dagg-OST000a-osc-ffff88006bb84800: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881577282200 Mar 16 03:19:03 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881577282200 Mar 16 03:19:03 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88279a5d2600 Mar 16 03:19:03 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88279a5d2600 Mar 16 03:19:03 john33 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:03 john33 kernel: Lustre: dagg-OST000a-osc-ffff88006bb84800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:19:03 john33 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:03 john32 kernel: Lustre: 928:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff882182fbef00 x1594699396464000/t0(0) o4->dagg-OST000d-osc-ffff8817c5384000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:03 john32 kernel: Lustre: 928:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:03 john32 kernel: Lustre: dagg-OST000d-osc-ffff8817c5384000: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880ffc31a000 Mar 16 03:19:03 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880ffc31a000 Mar 16 03:19:03 john32 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:03 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a426e00 Mar 16 03:19:03 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a426e00 Mar 16 03:19:03 john33 kernel: LNetError: 903:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594699393977312 length 1048576 too big: 1046348 left, 1046348 allowed Mar 16 03:19:03 john33 kernel: LNetError: 903:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:03 john32 kernel: Lustre: dagg-OST000d-osc-ffff8817c5384000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:03 john32 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:03 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882d8ea77200 Mar 16 03:19:03 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882d8ea77200 Mar 16 03:19:03 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88230b032e00 Mar 16 03:19:03 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88230b032e00 Mar 16 03:19:03 john32 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594699396560192 length 1048576 too big: 1048540 left, 1048540 allowed Mar 16 03:19:03 john32 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:03 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ae0cee600 Mar 16 03:19:03 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ae0cee600 Mar 16 03:19:03 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88158d996a00 Mar 16 03:19:03 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88158d996a00 Mar 16 03:19:03 john29 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff882c3e222d00 x1594699419066272/t0(0) o4->dagg-OST0005-osc-ffff88189afee000@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:19:03 john29 kernel: Lustre: dagg-OST0002-osc-ffff88189afee000: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 john29 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:03 john29 kernel: Lustre: dagg-OST0002-osc-ffff88189afee000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:19:03 john29 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:03 john29 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594699419161056 length 1048576 too big: 1048152 left, 1048152 allowed Mar 16 03:19:03 john29 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:03 john29 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:03 john13 kernel: Lustre: 935:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff88283c641800 x1594448319199040/t0(0) o4->dagg-OST000d-osc-ffff88015a2c6800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:03 john13 kernel: Lustre: dagg-OST000d-osc-ffff88015a2c6800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 john13 kernel: Lustre: dagg-OST000d-osc-ffff88015a2c6800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:03 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88230b032e00 Mar 16 03:19:03 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88230b032e00 Mar 16 03:19:03 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8826ad3cda00 Mar 16 03:19:03 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8826ad3cda00 Mar 16 03:19:03 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88279a5d2600 Mar 16 03:19:03 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88279a5d2600 Mar 16 03:19:03 john9 kernel: Lustre: 935:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff88258cc91b00 x1594448323945264/t0(0) o4->dagg-OST0004-osc-ffff882fdd33b800@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:03 john9 kernel: Lustre: dagg-OST0004-osc-ffff882fdd33b800: Connection to dagg-OST0004 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:03 john9 kernel: Lustre: dagg-OST0004-osc-ffff882fdd33b800: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:19:03 john13 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448319310336 length 1048576 too big: 1046108 left, 1046108 allowed Mar 16 03:19:03 john13 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:03 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c7ffadc00 Mar 16 03:19:03 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c7ffadc00 Mar 16 03:19:03 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b88600 Mar 16 03:19:03 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b88600 Mar 16 03:19:03 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8816d3c73800 Mar 16 03:19:03 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8816d3c73800 Mar 16 03:19:03 john9 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448324070656 length 1048576 too big: 1044696 left, 1044696 allowed Mar 16 03:19:03 john9 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:03 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8826ad3cda00 Mar 16 03:19:03 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88279a5d2600 Mar 16 03:19:03 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88279a5d2600 Mar 16 03:19:03 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8826ad3cda00 Mar 16 03:19:03 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ae0cee600 Mar 16 03:19:03 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ae0cee600 Mar 16 03:19:03 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed84a00 Mar 16 03:19:03 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed84a00 Mar 16 03:19:03 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882239a52200 Mar 16 03:19:03 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882239a52200 Mar 16 03:19:03 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b88800 Mar 16 03:19:03 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b88800 Mar 16 03:19:03 john23 kernel: Lustre: 924:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff8814da225d00 x1594448282171968/t0(0) o4->dagg-OST0002-osc-ffff882fdbc22800@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:03 john23 kernel: Lustre: 924:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:03 john23 kernel: Lustre: dagg-OST0002-osc-ffff882fdbc22800: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:04 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed84a00 Mar 16 03:19:04 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed84a00 Mar 16 03:19:04 john23 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:04 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881165147a00 Mar 16 03:19:04 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880bafefd000 Mar 16 03:19:04 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881165147a00 Mar 16 03:19:04 john23 kernel: Lustre: dagg-OST0002-osc-ffff882fdbc22800: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:19:04 john23 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:04 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880bafefd000 Mar 16 03:19:04 john6 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130699/real 1521130699] req@ffff8803cff98300 x1594448323930016/t0(0) o4->dagg-OST0002-osc-ffff881899c73800@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130743 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:04 john6 kernel: Lustre: dagg-OST0000-osc-ffff881899c73800: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:04 john6 kernel: Lustre: dagg-OST0000-osc-ffff881899c73800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:19:04 john6 kernel: LNetError: 912:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448324046416 length 1048576 too big: 1047972 left, 1047972 allowed Mar 16 03:19:04 john6 kernel: LNetError: 912:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:04 john6 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:19:04 john23 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448282277136 length 1048576 too big: 1047668 left, 1047668 allowed Mar 16 03:19:04 john23 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:04 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881165144e00 Mar 16 03:19:04 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881768838200 Mar 16 03:19:04 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881768838200 Mar 16 03:19:04 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881165144e00 Mar 16 03:19:04 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a421800 Mar 16 03:19:04 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a421800 Mar 16 03:19:04 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed84a00 Mar 16 03:19:04 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed84a00 Mar 16 03:19:04 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880bafefe000 Mar 16 03:19:04 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881165144a00 Mar 16 03:19:04 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dc52f6200 Mar 16 03:19:04 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880583af8400 Mar 16 03:19:04 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880bafefe000 Mar 16 03:19:04 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dc52f6200 Mar 16 03:19:04 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881165144a00 Mar 16 03:19:04 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880583af8400 Mar 16 03:19:04 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881768838200 Mar 16 03:19:04 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881768838200 Mar 16 03:19:04 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881766fa3200 Mar 16 03:19:04 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881766fa3200 Mar 16 03:19:04 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881768838200 Mar 16 03:19:04 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881768838200 Mar 16 03:19:04 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881766fa3200 Mar 16 03:19:04 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881766fa3200 Mar 16 03:19:04 john31 kernel: Lustre: 936:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130700/real 1521130700] req@ffff882758f22a00 x1594699394168800/t0(0) o4->dagg-OST0003-osc-ffff882fde9fd000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130744 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:04 john31 kernel: Lustre: dagg-OST0003-osc-ffff882fde9fd000: Connection to dagg-OST0003 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:04 john31 kernel: Lustre: dagg-OST0003-osc-ffff882fde9fd000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:19:04 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef5cc4200 Mar 16 03:19:04 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef5cc4200 Mar 16 03:19:04 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8ec00 Mar 16 03:19:04 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8ec00 Mar 16 03:19:04 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882224993800 Mar 16 03:19:04 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882224993800 Mar 16 03:19:04 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac956800 Mar 16 03:19:04 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac956800 Mar 16 03:19:04 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817deebae00 Mar 16 03:19:04 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817deebae00 Mar 16 03:19:04 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed84a00 Mar 16 03:19:04 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed84a00 Mar 16 03:19:04 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef5cc1e00 Mar 16 03:19:04 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef5cc1e00 Mar 16 03:19:04 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a426600 Mar 16 03:19:04 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a426600 Mar 16 03:19:04 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c7ffaea00 Mar 16 03:19:04 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c7ffaea00 Mar 16 03:19:04 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e21a6a200 Mar 16 03:19:04 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e21a6a200 Mar 16 03:19:04 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef5cc1e00 Mar 16 03:19:04 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef5cc1e00 Mar 16 03:19:04 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817deeba200 Mar 16 03:19:04 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817deeba200 Mar 16 03:19:05 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef5cc1e00 Mar 16 03:19:05 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef5cc1e00 Mar 16 03:19:05 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880710ec3000 Mar 16 03:19:05 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880710ec3000 Mar 16 03:19:05 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c7ffaec00 Mar 16 03:19:05 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c7ffaec00 Mar 16 03:19:05 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a426800 Mar 16 03:19:05 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a426800 Mar 16 03:19:05 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8821a45abe00 Mar 16 03:19:05 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8821a45abe00 Mar 16 03:19:05 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880583af8400 Mar 16 03:19:05 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880583af8400 Mar 16 03:19:05 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8ec00 Mar 16 03:19:05 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8ec00 Mar 16 03:19:05 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a426a00 Mar 16 03:19:05 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a426a00 Mar 16 03:19:05 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e6bc5fa00 Mar 16 03:19:05 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e6bc5fa00 Mar 16 03:19:05 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a425400 Mar 16 03:19:05 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a425400 Mar 16 03:19:06 john10 kernel: Lustre: 943:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130702/real 1521130702] req@ffff882063fff800 x1594448319882256/t0(0) o4->dagg-OST0001-osc-ffff882fd4b83800@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130746 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:06 john10 kernel: Lustre: 943:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:19:06 john10 kernel: Lustre: dagg-OST0001-osc-ffff882fd4b83800: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:06 john10 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:19:06 john10 kernel: Lustre: dagg-OST0001-osc-ffff882fd4b83800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:19:06 john10 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:19:06 john10 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448319982048 length 1048576 too big: 1046492 left, 1046492 allowed Mar 16 03:19:06 john10 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:06 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817deeba200 Mar 16 03:19:06 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817deeba200 Mar 16 03:19:06 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881766fa4c00 Mar 16 03:19:06 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881766fa4c00 Mar 16 03:19:06 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8826ad3cc600 Mar 16 03:19:06 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8826ad3cc600 Mar 16 03:19:06 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8ec00 Mar 16 03:19:06 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8ec00 Mar 16 03:19:06 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ae0cee600 Mar 16 03:19:06 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ae0cee600 Mar 16 03:19:06 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177f58f600 Mar 16 03:19:06 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177f58f600 Mar 16 03:19:06 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8821a45abe00 Mar 16 03:19:06 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8821a45abe00 Mar 16 03:19:06 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8816ce318600 Mar 16 03:19:06 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8816ce318600 Mar 16 03:19:06 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882065dd7400 Mar 16 03:19:06 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882065dd7400 Mar 16 03:19:07 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881fb155b800 Mar 16 03:19:07 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881fb155b800 Mar 16 03:19:07 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177f58b400 Mar 16 03:19:07 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177f58b400 Mar 16 03:19:07 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed84a00 Mar 16 03:19:07 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed84a00 Mar 16 03:19:07 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8821a45ac000 Mar 16 03:19:07 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8821a45ac000 Mar 16 03:19:08 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882bba6df800 Mar 16 03:19:08 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882bba6df800 Mar 16 03:19:08 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817deebce00 Mar 16 03:19:08 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817deebce00 Mar 16 03:19:08 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c7ffad400 Mar 16 03:19:08 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c7ffad400 Mar 16 03:19:08 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef5cc1200 Mar 16 03:19:08 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef5cc1200 Mar 16 03:19:09 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ecb6e7200 Mar 16 03:19:09 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ecb6e7200 Mar 16 03:19:09 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880710ec5400 Mar 16 03:19:09 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880710ec5400 Mar 16 03:19:10 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894ae600 Mar 16 03:19:10 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894ae600 Mar 16 03:19:10 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8821a45aa200 Mar 16 03:19:10 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8821a45aa200 Mar 16 03:19:10 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8821a45aa200 Mar 16 03:19:10 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8821a45aa200 Mar 16 03:19:10 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8821a45aa200 Mar 16 03:19:10 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8821a45aa200 Mar 16 03:19:11 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef5cc3800 Mar 16 03:19:11 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef5cc3800 Mar 16 03:19:11 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88290014cc00 Mar 16 03:19:11 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88290014cc00 Mar 16 03:19:12 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8816c75f2800 Mar 16 03:19:12 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8816c75f2800 Mar 16 03:19:12 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894ae400 Mar 16 03:19:12 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894ae400 Mar 16 03:19:14 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed86600 Mar 16 03:19:14 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed86600 Mar 16 03:19:15 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c7ffab000 Mar 16 03:19:15 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c7ffab000 Mar 16 03:19:15 john19 kernel: Lustre: 908:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130711/real 1521130711] req@ffff881d29064b00 x1594448327445664/t0(0) o4->dagg-OST000d-osc-ffff88189a323000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130755 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:15 john19 kernel: Lustre: 908:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:19:15 john19 kernel: Lustre: dagg-OST000d-osc-ffff88189a323000: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:15 john19 kernel: Lustre: dagg-OST000d-osc-ffff88189a323000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:23 john26 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448281435008 length 1048576 too big: 1047960 left, 1047960 allowed Mar 16 03:19:23 john26 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:23 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8811eee46e00 Mar 16 03:19:23 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8811eee46e00 Mar 16 03:19:23 john11 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130719/real 1521130719] req@ffff8826ea14ad00 x1594448320168128/t0(0) o4->dagg-OST000a-osc-ffff882fd571f000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130763 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:23 john11 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:19:23 john11 kernel: Lustre: dagg-OST000a-osc-ffff882fd571f000: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:23 john11 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:23 arkle6 kernel: Lustre: dagg-OST000a: Client 7c2b1b78-121a-b208-fc8a-57295d42d7f7 (at 192.168.44.111@o2ib44) reconnecting Mar 16 03:19:23 arkle6 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:19:23 arkle6 kernel: Lustre: dagg-OST000a: Connection restored to 9e90df21-9b59-537c-f486-0efeb807faad (at 192.168.44.111@o2ib44) Mar 16 03:19:23 arkle6 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:19:23 john11 kernel: Lustre: dagg-OST000a-osc-ffff882fd571f000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:19:23 john11 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:23 john11 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448320294032 length 1048576 too big: 1047284 left, 1047284 allowed Mar 16 03:19:23 john11 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:23 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8828303db000 Mar 16 03:19:23 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8828303db000 Mar 16 03:19:23 john25 kernel: LNetError: 903:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448280991216 length 1048576 too big: 1046340 left, 1046340 allowed Mar 16 03:19:23 john25 kernel: LNetError: 903:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:23 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881fb155e600 Mar 16 03:19:23 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881fb155e600 Mar 16 03:19:24 john12 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130720/real 1521130720] req@ffff881f017c1800 x1594448324471616/t0(0) o4->dagg-OST000e-osc-ffff88015a3ab800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130764 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:24 john12 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:19:24 john12 kernel: Lustre: dagg-OST000e-osc-ffff88015a3ab800: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:24 john12 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:24 john12 kernel: Lustre: dagg-OST000e-osc-ffff88015a3ab800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:19:24 john12 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:24 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881fb155be00 Mar 16 03:19:24 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881fb155be00 Mar 16 03:19:24 john31 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130720/real 1521130720] req@ffff881c56f45100 x1594699394190000/t0(0) o4->dagg-OST000e-osc-ffff882fde9fd000@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130764 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:24 john31 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:19:24 john31 kernel: Lustre: dagg-OST000e-osc-ffff882fde9fd000: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:24 john31 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:24 john31 kernel: Lustre: dagg-OST000e-osc-ffff882fde9fd000: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:19:24 john31 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:24 john31 kernel: LNetError: 895:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594699394285776 length 1048576 too big: 1048208 left, 1048208 allowed Mar 16 03:19:24 john31 kernel: LNetError: 895:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:24 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881fb155e000 Mar 16 03:19:24 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881fb155e000 Mar 16 03:19:25 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e832b1600 Mar 16 03:19:25 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177f58d200 Mar 16 03:19:25 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e832b1600 Mar 16 03:19:25 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177f58d200 Mar 16 03:19:25 john15 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448324055280 length 1048576 too big: 1046392 left, 1046392 allowed Mar 16 03:19:25 john15 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:26 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881fb155d000 Mar 16 03:19:26 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881fb155d000 Mar 16 03:19:26 arkle3 kernel: Lustre: dagg-OST0004: Client 7c2b1b78-121a-b208-fc8a-57295d42d7f7 (at 192.168.44.111@o2ib44) reconnecting Mar 16 03:19:26 arkle3 kernel: Lustre: Skipped 12 previous similar messages Mar 16 03:19:26 arkle3 kernel: Lustre: dagg-OST0004: Connection restored to 9e90df21-9b59-537c-f486-0efeb807faad (at 192.168.44.111@o2ib44) Mar 16 03:19:26 arkle3 kernel: Lustre: Skipped 12 previous similar messages Mar 16 03:19:26 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8816d3c73a00 Mar 16 03:19:26 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8816d3c73a00 Mar 16 03:19:26 arkle3 kernel: LustreError: 187521:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881045b59450 x1594448320177872/t0(0) o4->7c2b1b78-121a-b208-fc8a-57295d42d7f7@192.168.44.111@o2ib44:599/0 lens 608/448 e 0 to 0 dl 1521130809 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:19:26 arkle3 kernel: LustreError: 187521:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 30 previous similar messages Mar 16 03:19:26 arkle3 kernel: Lustre: dagg-OST0004: Bulk IO write error with 7c2b1b78-121a-b208-fc8a-57295d42d7f7 (at 192.168.44.111@o2ib44), client will retry: rc = -110 Mar 16 03:19:26 arkle3 kernel: Lustre: Skipped 30 previous similar messages Mar 16 03:19:46 john25 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130742/real 1521130742] req@ffff881574af9e00 x1594448280872016/t0(0) o4->dagg-OST0006-osc-ffff882fd7db5800@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130786 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:46 john25 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:46 john25 kernel: Lustre: dagg-OST0006-osc-ffff882fd7db5800: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:46 john14 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130742/real 1521130742] req@ffff880a99ed1200 x1594448317087008/t0(0) o4->dagg-OST0006-osc-ffff88189a368800@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130786 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:19:46 john14 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:19:46 john14 kernel: Lustre: dagg-OST0006-osc-ffff88189a368800: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:46 john25 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:46 john14 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:46 arkle4 kernel: Lustre: dagg-OST0006: Client 40d9b451-fab3-be40-89fb-0c9b04087671 (at 192.168.44.125@o2ib44) reconnecting Mar 16 03:19:46 arkle4 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:19:46 arkle4 kernel: Lustre: dagg-OST0006: Connection restored to edb2edda-7927-928c-4497-0706e1cfa6ce (at 192.168.44.125@o2ib44) Mar 16 03:19:46 arkle4 kernel: Lustre: Skipped 14 previous similar messages Mar 16 03:19:46 john14 kernel: Lustre: dagg-OST0006-osc-ffff88189a368800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:19:46 john14 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:46 john25 kernel: Lustre: dagg-OST0006-osc-ffff882fd7db5800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:19:46 john25 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:46 john14 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448317236432 length 1048576 too big: 1045352 left, 1045352 allowed Mar 16 03:19:46 john14 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:46 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882065dd5200 Mar 16 03:19:46 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882065dd5200 Mar 16 03:19:46 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8e400 Mar 16 03:19:46 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e093a00 Mar 16 03:19:46 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8e400 Mar 16 03:19:46 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e093a00 Mar 16 03:19:46 john30 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130742/real 1521130742] req@ffff882e4338a400 x1594699400481984/t0(0) o4->dagg-OST000c-osc-ffff88189ac92000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130786 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:46 john30 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:46 john30 kernel: Lustre: dagg-OST000c-osc-ffff88189ac92000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:46 john30 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:46 john28 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130742/real 1521130742] req@ffff881aaab92d00 x1594448282246752/t0(0) o4->dagg-OST0006-osc-ffff8817d9cf4000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130786 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:46 arkle7 kernel: Lustre: dagg-OST000c: Client faed83b4-3a5e-2a8e-2882-7e99d174dfb0 (at 192.168.44.130@o2ib44) reconnecting Mar 16 03:19:46 arkle7 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:46 arkle7 kernel: Lustre: dagg-OST000c: Connection restored to 2beb6461-856d-b027-2da8-040c1c990221 (at 192.168.44.130@o2ib44) Mar 16 03:19:46 arkle7 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:46 john30 kernel: Lustre: dagg-OST000c-osc-ffff88189ac92000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:46 john30 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:46 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507701a00 Mar 16 03:19:46 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507701a00 Mar 16 03:19:46 john30 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594699400619632 length 1048576 too big: 1048516 left, 1048516 allowed Mar 16 03:19:46 john30 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:46 john27 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130742/real 1521130742] req@ffff881b4e2d3600 x1594448285369712/t0(0) o4->dagg-OST000f-osc-ffff88015a1e4800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130786 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:46 john27 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:46 john20 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130742/real 1521130742] req@ffff8828a2b52100 x1594448327525968/t0(0) o4->dagg-OST000d-osc-ffff8817da6ad800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130786 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:46 john20 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:46 john20 kernel: Lustre: dagg-OST000d-osc-ffff8817da6ad800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:46 john27 kernel: Lustre: dagg-OST000f-osc-ffff88015a1e4800: Connection to dagg-OST000f (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:46 john27 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:46 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882900148400 Mar 16 03:19:46 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882900148400 Mar 16 03:19:46 arkle8 kernel: Lustre: dagg-OST000f: Client 219d9f55-42f0-6656-ab6f-f4de9ef8c6c7 (at 192.168.44.127@o2ib44) reconnecting Mar 16 03:19:46 arkle8 kernel: Lustre: Skipped 18 previous similar messages Mar 16 03:19:46 arkle8 kernel: Lustre: dagg-OST000f: Connection restored to c8b7715d-04bb-a76a-3db9-c45127314e23 (at 192.168.44.127@o2ib44) Mar 16 03:19:46 john20 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:46 john20 kernel: Lustre: dagg-OST000d-osc-ffff8817da6ad800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:46 arkle8 kernel: Lustre: Skipped 12 previous similar messages Mar 16 03:19:46 john20 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:46 john27 kernel: Lustre: dagg-OST000f-osc-ffff88015a1e4800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:19:46 john27 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:46 arkle4 kernel: LustreError: 302876:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882f65b65450 x1594448280872016/t0(0) o4->40d9b451-fab3-be40-89fb-0c9b04087671@192.168.44.125@o2ib44:619/0 lens 608/448 e 0 to 0 dl 1521130829 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:19:46 arkle4 kernel: Lustre: dagg-OST0006: Bulk IO write error with 23017e25-beb4-2d0c-1e7b-9a1bd7cbb415 (at 192.168.44.114@o2ib44), client will retry: rc = -110 Mar 16 03:19:46 arkle4 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:46 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881751ca5600 Mar 16 03:19:46 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881751ca5600 Mar 16 03:19:46 arkle4 kernel: LustreError: 302876:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 15 previous similar messages Mar 16 03:19:46 john20 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448327693744 length 1048576 too big: 1045364 left, 1045364 allowed Mar 16 03:19:46 john20 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:46 john27 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448285507168 length 1048576 too big: 1046096 left, 1046096 allowed Mar 16 03:19:46 john27 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:46 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882900148400 Mar 16 03:19:46 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882900148400 Mar 16 03:19:46 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177f58bc00 Mar 16 03:19:46 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177f58bc00 Mar 16 03:19:46 arkle8 kernel: LustreError: 448809:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88062798a050 x1594448285369712/t0(0) o4->219d9f55-42f0-6656-ab6f-f4de9ef8c6c7@192.168.44.127@o2ib44:619/0 lens 608/448 e 0 to 0 dl 1521130829 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:19:46 john28 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:46 john28 kernel: Lustre: dagg-OST0006-osc-ffff8817d9cf4000: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:46 john28 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:46 john28 kernel: Lustre: dagg-OST0006-osc-ffff8817d9cf4000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:19:46 john28 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:46 arkle8 kernel: LustreError: 448809:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 20 previous similar messages Mar 16 03:19:46 arkle8 kernel: Lustre: dagg-OST000f: Bulk IO write error with 219d9f55-42f0-6656-ab6f-f4de9ef8c6c7 (at 192.168.44.127@o2ib44), client will retry: rc = -110 Mar 16 03:19:46 arkle8 kernel: Lustre: Skipped 20 previous similar messages Mar 16 03:19:46 john15 kernel: Lustre: 949:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130742/real 1521130742] req@ffff88207ef11200 x1594448323936064/t0(0) o4->dagg-OST000d-osc-ffff882fde9f4800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130786 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:46 john15 kernel: Lustre: 949:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:46 john15 kernel: Lustre: dagg-OST000d-osc-ffff882fde9f4800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8801ebc8bc00 Mar 16 03:19:47 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8801ebc8bc00 Mar 16 03:19:47 john15 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 john21 kernel: Lustre: 918:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130742/real 1521130742] req@ffff8813c2e68300 x1594448332914000/t0(0) o4->dagg-OST000c-osc-ffff882fc5588800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130786 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:47 john21 kernel: Lustre: 918:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:19:47 john21 kernel: Lustre: dagg-OST000c-osc-ffff882fc5588800: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 john15 kernel: Lustre: dagg-OST000d-osc-ffff882fde9f4800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:47 john15 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 john21 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507701a00 Mar 16 03:19:47 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882900148400 Mar 16 03:19:47 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882900148400 Mar 16 03:19:47 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507701a00 Mar 16 03:19:47 john24 kernel: Lustre: 951:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff882c67f3f500 x1594448284013248/t0(0) o4->dagg-OST000c-osc-ffff882fcb645000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:47 john24 kernel: Lustre: 951:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:47 john24 kernel: Lustre: dagg-OST000c-osc-ffff882fcb645000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 john24 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507701a00 Mar 16 03:19:47 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507701a00 Mar 16 03:19:47 john24 kernel: Lustre: dagg-OST000c-osc-ffff882fcb645000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:47 john24 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177f58bc00 Mar 16 03:19:47 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177f58bc00 Mar 16 03:19:47 john21 kernel: Lustre: dagg-OST000c-osc-ffff882fc5588800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:47 john21 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 john21 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448333033856 length 1048576 too big: 1044712 left, 1044712 allowed Mar 16 03:19:47 john21 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:47 john22 kernel: Lustre: 923:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff88271d3a6000 x1594448280687824/t0(0) o4->dagg-OST0007-osc-ffff88189a897000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:47 john22 kernel: Lustre: dagg-OST000c-osc-ffff88189a897000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 john22 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 john22 kernel: Lustre: dagg-OST000c-osc-ffff88189a897000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:47 john22 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 john22 kernel: LNetError: 904:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448280825232 length 1048576 too big: 1048436 left, 1048436 allowed Mar 16 03:19:47 john22 kernel: LNetError: 904:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:47 john22 kernel: Lustre: 923:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:47 john24 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448284152464 length 1048576 too big: 1046748 left, 1046748 allowed Mar 16 03:19:47 john24 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:47 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b88c00 Mar 16 03:19:47 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b88c00 Mar 16 03:19:47 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507701a00 Mar 16 03:19:47 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507701a00 Mar 16 03:19:47 john17 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff880e00b31500 x1594448316853392/t0(0) o4->dagg-OST0002-osc-ffff881899e5b000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:47 john17 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:47 john17 kernel: Lustre: dagg-OST0002-osc-ffff881899e5b000: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8828303d8c00 Mar 16 03:19:47 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88160df4fe00 Mar 16 03:19:47 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8828303d8c00 Mar 16 03:19:47 john17 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88160df4fe00 Mar 16 03:19:47 arkle2 kernel: Lustre: dagg-OST0002: Client 6b4209e2-620e-030d-e2d1-b9c575162077 (at 192.168.44.117@o2ib44) reconnecting Mar 16 03:19:47 arkle2 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:19:47 arkle2 kernel: Lustre: dagg-OST0002: Connection restored to 7002ccab-a112-d1d5-6fdf-c8c42ad342fc (at 192.168.44.117@o2ib44) Mar 16 03:19:47 arkle6 kernel: LustreError: 199667:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88101bd91050 x1594448327431632/t0(0) o4->d4acd478-18fd-71d3-c177-089c607f8639@192.168.44.119@o2ib44:620/0 lens 608/448 e 0 to 0 dl 1521130830 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:19:47 arkle2 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:19:47 john17 kernel: Lustre: dagg-OST0002-osc-ffff881899e5b000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:19:47 john17 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 arkle6 kernel: Lustre: dagg-OST000a: Bulk IO write error with d4acd478-18fd-71d3-c177-089c607f8639 (at 192.168.44.119@o2ib44), client will retry: rc = -110 Mar 16 03:19:47 arkle6 kernel: Lustre: Skipped 16 previous similar messages Mar 16 03:19:47 arkle6 kernel: LustreError: 199667:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 17 previous similar messages Mar 16 03:19:47 john19 kernel: Lustre: 941:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff880808246c00 x1594448327426912/t0(0) o4->dagg-OST000a-osc-ffff88189a323000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:19:47 john19 kernel: Lustre: dagg-OST000a-osc-ffff88189a323000: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 john19 kernel: Lustre: dagg-OST000a-osc-ffff88189a323000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:19:47 john19 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448327577504 length 1048576 too big: 1045432 left, 1045432 allowed Mar 16 03:19:47 john19 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:47 john19 kernel: Lustre: 941:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:19:47 john17 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448316996032 length 1048576 too big: 1044508 left, 1044508 allowed Mar 16 03:19:47 john17 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:47 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8828303dd200 Mar 16 03:19:47 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8828303dd200 Mar 16 03:19:47 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8821a45af600 Mar 16 03:19:47 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8821a45af600 Mar 16 03:19:47 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef5cc6800 Mar 16 03:19:47 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef5cc6800 Mar 16 03:19:47 arkle2 kernel: LustreError: 350069:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff8828e5864c50 x1594448316853392/t0(0) o4->6b4209e2-620e-030d-e2d1-b9c575162077@192.168.44.117@o2ib44:620/0 lens 608/448 e 0 to 0 dl 1521130830 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:19:47 arkle2 kernel: LustreError: 350069:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 15 previous similar messages Mar 16 03:19:47 arkle2 kernel: Lustre: dagg-OST0002: Bulk IO write error with 6b4209e2-620e-030d-e2d1-b9c575162077 (at 192.168.44.117@o2ib44), client will retry: rc = -110 Mar 16 03:19:47 arkle2 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:47 john26 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff882a53e52700 x1594448281318416/t0(0) o4->dagg-OST0008-osc-ffff882fde832000@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:47 john26 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:47 john26 kernel: Lustre: dagg-OST0008-osc-ffff882fde832000: Connection to dagg-OST0008 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e1b695a00 Mar 16 03:19:47 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e1b695a00 Mar 16 03:19:47 john26 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 arkle5 kernel: Lustre: dagg-OST0008: Client f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44) reconnecting Mar 16 03:19:47 arkle5 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:47 arkle5 kernel: Lustre: dagg-OST0008: Connection restored to d5de0e2d-4431-41fa-738a-2fa7d0a3c365 (at 192.168.44.126@o2ib44) Mar 16 03:19:47 arkle5 kernel: Lustre: Skipped 15 previous similar messages Mar 16 03:19:47 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8ee00 Mar 16 03:19:47 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8ee00 Mar 16 03:19:47 arkle1 kernel: Lustre: dagg-OST0000: Client 6ec8b14c-8574-1b1c-979a-d89152f9126c (at 192.168.44.124@o2ib44) reconnecting Mar 16 03:19:47 arkle1 kernel: Lustre: Skipped 14 previous similar messages Mar 16 03:19:47 arkle1 kernel: Lustre: dagg-OST0000: Connection restored to 8f94461e-6f3c-b18c-f4e9-0d4fa2a51df8 (at 192.168.44.124@o2ib44) Mar 16 03:19:47 john26 kernel: Lustre: dagg-OST0008-osc-ffff882fde832000: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:19:47 john26 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 john18 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff882b86330900 x1594448284367264/t0(0) o4->dagg-OST0000-osc-ffff88015a243000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:47 john18 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:47 john18 kernel: Lustre: dagg-OST0000-osc-ffff88015a243000: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894a8a00 Mar 16 03:19:47 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894a8a00 Mar 16 03:19:47 arkle1 kernel: Lustre: Skipped 14 previous similar messages Mar 16 03:19:47 john16 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff881e67300f00 x1594448321487984/t0(0) o4->dagg-OST0007-osc-ffff882fc87e8000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:19:47 john16 kernel: Lustre: dagg-OST0007-osc-ffff882fc87e8000: Connection to dagg-OST0007 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 john16 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 john16 kernel: Lustre: dagg-OST0007-osc-ffff882fc87e8000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:19:47 john16 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 john16 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 16 03:19:47 john18 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:47 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed82600 Mar 16 03:19:47 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed82600 Mar 16 03:19:47 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88164874fc00 Mar 16 03:19:47 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88164874fc00 Mar 16 03:19:47 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8ee00 Mar 16 03:19:47 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8ee00 Mar 16 03:19:47 john18 kernel: Lustre: dagg-OST0000-osc-ffff88015a243000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:19:47 john18 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:47 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88164874fc00 Mar 16 03:19:47 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88164874fc00 Mar 16 03:19:47 john18 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448284511792 length 1048576 too big: 1048556 left, 1048556 allowed Mar 16 03:19:47 john18 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:47 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8801ebc8bc00 Mar 16 03:19:47 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8801ebc8bc00 Mar 16 03:19:47 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881dd4163000 Mar 16 03:19:47 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881dd4163000 Mar 16 03:19:47 john8 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff88270812b300 x1594448316803760/t0(0) o4->dagg-OST0000-osc-ffff88015a2fe000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:47 john8 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:19:47 john8 kernel: Lustre: dagg-OST0000-osc-ffff88015a2fe000: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c0aeef600 Mar 16 03:19:47 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c0aeef600 Mar 16 03:19:47 john8 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:47 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894a8a00 Mar 16 03:19:47 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894a8a00 Mar 16 03:19:47 john8 kernel: Lustre: dagg-OST0000-osc-ffff88015a2fe000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:19:47 john8 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:47 john8 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448316925984 length 1048576 too big: 1047284 left, 1047284 allowed Mar 16 03:19:47 john8 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:47 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ec3e0d600 Mar 16 03:19:47 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ec3e0d600 Mar 16 03:19:47 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881648748a00 Mar 16 03:19:47 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881648748a00 Mar 16 03:19:47 john33 kernel: Lustre: 940:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff882e3f367500 x1594699393878992/t0(0) o4->dagg-OST000a-osc-ffff88006bb84800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:47 john33 kernel: Lustre: 940:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:47 john33 kernel: Lustre: dagg-OST000a-osc-ffff88006bb84800: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 john33 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 john29 kernel: Lustre: 936:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff882c3e225a00 x1594699419066352/t0(0) o4->dagg-OST0005-osc-ffff88189afee000@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:47 john29 kernel: Lustre: 936:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:19:47 john29 kernel: Lustre: dagg-OST0005-osc-ffff88189afee000: Connection to dagg-OST0005 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 john33 kernel: Lustre: dagg-OST000a-osc-ffff88006bb84800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:19:47 john33 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8828303dd200 Mar 16 03:19:47 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8828303dd200 Mar 16 03:19:47 john32 kernel: Lustre: 928:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff882182fbef00 x1594699396464000/t0(0) o4->dagg-OST000d-osc-ffff8817c5384000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:47 john32 kernel: Lustre: 928:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:47 john32 kernel: Lustre: dagg-OST000d-osc-ffff8817c5384000: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 john29 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:47 john32 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:47 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817583a2e00 Mar 16 03:19:47 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e7ac1f600 Mar 16 03:19:47 john29 kernel: Lustre: dagg-OST0002-osc-ffff88189afee000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:19:47 john29 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:19:47 john33 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594699394018320 length 1048576 too big: 1046348 left, 1046348 allowed Mar 16 03:19:47 john33 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:47 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817583a2e00 Mar 16 03:19:47 john32 kernel: Lustre: dagg-OST000d-osc-ffff8817c5384000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:19:47 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e7ac1f600 Mar 16 03:19:47 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880923832a00 Mar 16 03:19:47 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880923832a00 Mar 16 03:19:47 john29 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594699419184720 length 1048576 too big: 1048152 left, 1048152 allowed Mar 16 03:19:47 john29 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:47 john32 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:47 john32 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594699396601184 length 1048576 too big: 1048540 left, 1048540 allowed Mar 16 03:19:47 john32 kernel: LNetError: 898:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:47 john13 kernel: Lustre: 935:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff88283c641800 x1594448319199040/t0(0) o4->dagg-OST000d-osc-ffff88015a2c6800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:47 john13 kernel: Lustre: 935:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:47 john13 kernel: Lustre: dagg-OST000d-osc-ffff88015a2c6800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894a8a00 Mar 16 03:19:47 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894a8a00 Mar 16 03:19:47 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880bafefd400 Mar 16 03:19:47 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880bafefd400 Mar 16 03:19:47 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817583a2e00 Mar 16 03:19:47 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e7ac1f600 Mar 16 03:19:47 john13 kernel: Lustre: dagg-OST0007-osc-ffff88015a2c6800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:19:47 john13 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:47 john13 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448319334128 length 1048576 too big: 1048108 left, 1048108 allowed Mar 16 03:19:47 john13 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:47 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507704000 Mar 16 03:19:47 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507704000 Mar 16 03:19:47 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8ec00 Mar 16 03:19:47 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8ec00 Mar 16 03:19:47 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817583a2e00 Mar 16 03:19:47 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e7ac1f600 Mar 16 03:19:47 john9 kernel: Lustre: 935:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff88258cc91b00 x1594448323945264/t0(0) o4->dagg-OST0004-osc-ffff882fdd33b800@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:47 john9 kernel: Lustre: 935:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:47 john9 kernel: Lustre: dagg-OST0004-osc-ffff882fdd33b800: Connection to dagg-OST0004 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:47 john9 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 john9 kernel: Lustre: dagg-OST0004-osc-ffff882fdd33b800: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:19:47 john13 kernel: Lustre: Skipped 6 previous similar messages Mar 16 03:19:47 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8828303dd200 Mar 16 03:19:47 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8828303dd200 Mar 16 03:19:47 john9 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:47 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507704000 Mar 16 03:19:47 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507704000 Mar 16 03:19:47 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817583a2e00 Mar 16 03:19:47 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817583a2e00 Mar 16 03:19:47 john9 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448324094704 length 1048576 too big: 1044696 left, 1044696 allowed Mar 16 03:19:47 john9 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:47 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894a8a00 Mar 16 03:19:47 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894a8a00 Mar 16 03:19:47 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88157ed82600 Mar 16 03:19:47 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88157ed82600 Mar 16 03:19:47 john7 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff8817a06ff500 x1594448319400640/t0(0) o4->dagg-OST0006-osc-ffff882fda4b4000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:47 john7 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:47 john7 kernel: Lustre: dagg-OST0006-osc-ffff882fda4b4000: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:48 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88169cc5b200 Mar 16 03:19:48 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88169cc5b200 Mar 16 03:19:48 john7 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:48 john6 kernel: Lustre: 954:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130743/real 1521130743] req@ffff8819dca5c200 x1594448323927280/t0(0) o4->dagg-OST0000-osc-ffff881899c73800@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130787 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:48 john6 kernel: Lustre: 954:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:48 john6 kernel: Lustre: dagg-OST0000-osc-ffff881899c73800: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:48 john7 kernel: Lustre: dagg-OST0006-osc-ffff882fda4b4000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:19:48 john6 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:48 john7 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:48 john6 kernel: Lustre: dagg-OST0000-osc-ffff881899c73800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:19:48 john6 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:48 john7 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448319545088 length 1048576 too big: 1045464 left, 1045464 allowed Mar 16 03:19:48 john7 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:48 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef5cc6800 Mar 16 03:19:48 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881165140800 Mar 16 03:19:48 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881165140800 Mar 16 03:19:48 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880bafefea00 Mar 16 03:19:48 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880bafefea00 Mar 16 03:19:48 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef5cc6800 Mar 16 03:19:48 john6 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448324070944 length 1048576 too big: 1047972 left, 1047972 allowed Mar 16 03:19:48 john6 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:48 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8801ebc8bc00 Mar 16 03:19:48 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8801ebc8bc00 Mar 16 03:19:48 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880598b85600 Mar 16 03:19:48 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880598b85600 Mar 16 03:19:48 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88164874f800 Mar 16 03:19:48 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88164874f800 Mar 16 03:19:48 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881d86a78600 Mar 16 03:19:48 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881d86a78600 Mar 16 03:19:48 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880598b85c00 Mar 16 03:19:48 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880ffc31b400 Mar 16 03:19:48 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f46287c00 Mar 16 03:19:48 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894ab000 Mar 16 03:19:48 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880598b85c00 Mar 16 03:19:48 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f46287c00 Mar 16 03:19:48 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880ffc31b400 Mar 16 03:19:48 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894ab000 Mar 16 03:19:48 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef5cc6c00 Mar 16 03:19:48 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef5cc6c00 Mar 16 03:19:48 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88169cc5e200 Mar 16 03:19:48 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88169cc5e200 Mar 16 03:19:48 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881648748600 Mar 16 03:19:48 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881648748600 Mar 16 03:19:48 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef5cc6c00 Mar 16 03:19:48 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef5cc6c00 Mar 16 03:19:48 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881648748c00 Mar 16 03:19:48 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881648748c00 Mar 16 03:19:48 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef5cc6c00 Mar 16 03:19:48 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef5cc6c00 Mar 16 03:19:48 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817809dda00 Mar 16 03:19:48 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817809dda00 Mar 16 03:19:48 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881f2765ee00 Mar 16 03:19:48 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881f2765ee00 Mar 16 03:19:48 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881f2765ee00 Mar 16 03:19:48 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881f2765ee00 Mar 16 03:19:48 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b89200 Mar 16 03:19:48 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b89200 Mar 16 03:19:48 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88169cc5ac00 Mar 16 03:19:48 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88169cc5ac00 Mar 16 03:19:48 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef5cc6c00 Mar 16 03:19:48 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef5cc6c00 Mar 16 03:19:48 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881d86a78600 Mar 16 03:19:48 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881d86a78600 Mar 16 03:19:48 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8801ebc8bc00 Mar 16 03:19:48 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8801ebc8bc00 Mar 16 03:19:48 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507704000 Mar 16 03:19:48 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507704000 Mar 16 03:19:48 john23 kernel: Lustre: 924:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130744/real 1521130744] req@ffff8814da225d00 x1594448282171968/t0(0) o4->dagg-OST0002-osc-ffff882fdbc22800@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130788 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:48 john23 kernel: Lustre: 924:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:19:48 john23 kernel: Lustre: dagg-OST0001-osc-ffff882fdbc22800: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:48 john23 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:48 john23 kernel: Lustre: dagg-OST0001-osc-ffff882fdbc22800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:19:48 john23 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:19:48 john23 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448282321168 length 1048576 too big: 1045728 left, 1045728 allowed Mar 16 03:19:48 john23 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:19:48 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dd0425e00 Mar 16 03:19:48 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dd0425e00 Mar 16 03:19:48 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88176883ee00 Mar 16 03:19:48 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88176883ee00 Mar 16 03:19:49 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ed2784200 Mar 16 03:19:49 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ed2784200 Mar 16 03:19:49 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88290014c200 Mar 16 03:19:49 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88290014c200 Mar 16 03:19:49 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507704000 Mar 16 03:19:49 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507704000 Mar 16 03:19:49 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881d86a78600 Mar 16 03:19:49 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881d86a78600 Mar 16 03:19:49 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f46285200 Mar 16 03:19:49 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f46285200 Mar 16 03:19:49 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817809da800 Mar 16 03:19:49 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817809da800 Mar 16 03:19:49 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8fa00 Mar 16 03:19:49 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8fa00 Mar 16 03:19:49 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881d86a78600 Mar 16 03:19:49 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881d86a78600 Mar 16 03:19:49 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880941094a00 Mar 16 03:19:49 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880941094a00 Mar 16 03:19:49 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881d86a78600 Mar 16 03:19:49 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881d86a78600 Mar 16 03:19:50 john10 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130746/real 1521130746] req@ffff881757b33300 x1594448319886272/t0(0) o4->dagg-OST000f-osc-ffff882fd4b83800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130790 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:19:50 john10 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:19:50 john10 kernel: Lustre: dagg-OST000f-osc-ffff882fd4b83800: Connection to dagg-OST000f (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:19:50 john10 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:50 john10 kernel: Lustre: dagg-OST000f-osc-ffff882fd4b83800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:19:50 john10 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:19:50 john10 kernel: LNetError: 912:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448320026080 length 1048576 too big: 1046464 left, 1046464 allowed Mar 16 03:19:50 john10 kernel: LNetError: 912:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:19:50 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac950200 Mar 16 03:19:50 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac950200 Mar 16 03:19:50 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880941094a00 Mar 16 03:19:50 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880941094a00 Mar 16 03:19:50 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88164874f800 Mar 16 03:19:50 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88164874f800 Mar 16 03:19:50 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817583a6800 Mar 16 03:19:50 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817583a6800 Mar 16 03:19:50 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8fa00 Mar 16 03:19:50 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8fa00 Mar 16 03:19:50 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8828303dd200 Mar 16 03:19:50 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8828303dd200 Mar 16 03:19:50 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817809d9600 Mar 16 03:19:50 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817809d9600 Mar 16 03:19:50 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882744641000 Mar 16 03:19:50 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882744641000 Mar 16 03:19:50 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8811eee40200 Mar 16 03:19:50 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8811eee40200 Mar 16 03:19:50 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882baed25000 Mar 16 03:19:50 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882baed25000 Mar 16 03:19:51 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8811eee40200 Mar 16 03:19:51 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8811eee40200 Mar 16 03:19:51 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88169cc5b800 Mar 16 03:19:51 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88169cc5b800 Mar 16 03:19:51 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817809d9c00 Mar 16 03:19:51 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817809d9c00 Mar 16 03:19:52 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a427800 Mar 16 03:19:52 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a427800 Mar 16 03:19:52 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac956000 Mar 16 03:19:52 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac956000 Mar 16 03:19:52 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507704000 Mar 16 03:19:52 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507704000 Mar 16 03:19:52 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9762000 Mar 16 03:19:52 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9762000 Mar 16 03:19:53 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a427800 Mar 16 03:19:53 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a427800 Mar 16 03:19:53 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88290014a000 Mar 16 03:19:53 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88290014a000 Mar 16 03:19:54 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c6c9ae600 Mar 16 03:19:54 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c6c9ae600 Mar 16 03:19:54 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882b4a523c00 Mar 16 03:19:54 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882b4a523c00 Mar 16 03:19:54 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882b4a523c00 Mar 16 03:19:54 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882b4a523c00 Mar 16 03:19:54 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882b4a523c00 Mar 16 03:19:54 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882b4a523c00 Mar 16 03:19:55 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9766600 Mar 16 03:19:55 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9766600 Mar 16 03:19:55 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88290014dc00 Mar 16 03:19:55 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88290014dc00 Mar 16 03:19:56 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882744645a00 Mar 16 03:19:56 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882744645a00 Mar 16 03:19:56 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c6c9aea00 Mar 16 03:19:56 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c6c9aea00 Mar 16 03:19:58 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88169cc5fc00 Mar 16 03:19:58 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88169cc5fc00 Mar 16 03:19:59 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507706a00 Mar 16 03:19:59 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507706a00 Mar 16 03:20:07 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8811eee46200 Mar 16 03:20:07 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8811eee46200 Mar 16 03:20:07 john11 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130763/real 1521130763] req@ffff8826ea14ad00 x1594448320168128/t0(0) o4->dagg-OST000a-osc-ffff882fd571f000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130807 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:07 john11 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:20:07 john11 kernel: Lustre: dagg-OST000a-osc-ffff882fd571f000: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:07 john11 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:20:07 john11 kernel: Lustre: dagg-OST000a-osc-ffff882fd571f000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:20:07 john11 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:20:07 john11 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448320334944 length 1048576 too big: 1047284 left, 1047284 allowed Mar 16 03:20:07 john11 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:20:07 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8828303dd200 Mar 16 03:20:07 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8828303dd200 Mar 16 03:20:07 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880941097e00 Mar 16 03:20:07 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880941097e00 Mar 16 03:20:08 john12 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130764/real 1521130764] req@ffff881f017c1800 x1594448324471616/t0(0) o4->dagg-OST000e-osc-ffff88015a3ab800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130808 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:08 john12 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:20:08 john12 kernel: Lustre: dagg-OST000e-osc-ffff88015a3ab800: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:08 john12 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:08 john12 kernel: Lustre: dagg-OST000e-osc-ffff88015a3ab800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:20:08 john12 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:08 john12 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448324607776 length 1048576 too big: 1046984 left, 1046984 allowed Mar 16 03:20:08 john12 kernel: LNetError: 908:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 6 previous similar messages Mar 16 03:20:08 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880941097e00 Mar 16 03:20:08 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880941097e00 Mar 16 03:20:08 john31 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130764/real 1521130764] req@ffff881c56f45100 x1594699394190000/t0(0) o4->dagg-OST000e-osc-ffff882fde9fd000@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130808 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:08 john31 kernel: Lustre: 925:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:20:08 john31 kernel: Lustre: dagg-OST000e-osc-ffff882fde9fd000: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:08 john31 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:08 john31 kernel: Lustre: dagg-OST000e-osc-ffff882fde9fd000: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:20:08 john31 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:08 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880941097e00 Mar 16 03:20:08 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880941097e00 Mar 16 03:20:09 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880941097e00 Mar 16 03:20:09 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880941097e00 Mar 16 03:20:09 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8811eee42a00 Mar 16 03:20:09 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e832b6000 Mar 16 03:20:09 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8811eee42a00 Mar 16 03:20:09 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e832b6000 Mar 16 03:20:10 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817583a4600 Mar 16 03:20:10 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817583a4600 Mar 16 03:20:30 john14 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130786/real 1521130786] req@ffff880a99ed1200 x1594448317087008/t0(0) o4->dagg-OST0006-osc-ffff88189a368800@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130830 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:20:30 john14 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:20:30 john14 kernel: Lustre: dagg-OST0006-osc-ffff88189a368800: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:30 john14 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:20:30 arkle4 kernel: Lustre: dagg-OST0006: Client 23017e25-beb4-2d0c-1e7b-9a1bd7cbb415 (at 192.168.44.114@o2ib44) reconnecting Mar 16 03:20:30 arkle4 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:20:30 arkle4 kernel: Lustre: dagg-OST0006: Connection restored to 78033fff-f4e3-2589-3920-d6d5c19d59a2 (at 192.168.44.114@o2ib44) Mar 16 03:20:30 arkle4 kernel: Lustre: Skipped 12 previous similar messages Mar 16 03:20:30 john14 kernel: Lustre: dagg-OST0006-osc-ffff88189a368800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:20:30 john14 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:20:30 john25 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448281037392 length 1048576 too big: 1048092 left, 1048092 allowed Mar 16 03:20:30 john25 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 5 previous similar messages Mar 16 03:20:30 john14 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448317263296 length 1048576 too big: 1047896 left, 1047896 allowed Mar 16 03:20:30 john14 kernel: LNetError: 909:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:20:30 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882baed24600 Mar 16 03:20:30 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882baed24600 Mar 16 03:20:30 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e094e00 Mar 16 03:20:30 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e094e00 Mar 16 03:20:30 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8f400 Mar 16 03:20:30 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8f400 Mar 16 03:20:30 john28 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448282439696 length 1048576 too big: 1045344 left, 1045344 allowed Mar 16 03:20:30 john28 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 9 previous similar messages Mar 16 03:20:30 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507707400 Mar 16 03:20:30 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507707400 Mar 16 03:20:30 arkle7 kernel: LustreError: 175488:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881791adb850 x1594448317081440/t0(0) o4->23017e25-beb4-2d0c-1e7b-9a1bd7cbb415@192.168.44.114@o2ib44:663/0 lens 608/448 e 0 to 0 dl 1521130873 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:20:30 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882baed24600 Mar 16 03:20:30 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882baed24600 Mar 16 03:20:30 john30 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130786/real 1521130786] req@ffff882e4338a400 x1594699400481984/t0(0) o4->dagg-OST000c-osc-ffff88189ac92000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130830 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:30 john30 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:20:30 john30 kernel: Lustre: dagg-OST000c-osc-ffff88189ac92000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:30 arkle7 kernel: LustreError: 175488:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 31 previous similar messages Mar 16 03:20:30 arkle7 kernel: Lustre: dagg-OST000c: Bulk IO write error with 23017e25-beb4-2d0c-1e7b-9a1bd7cbb415 (at 192.168.44.114@o2ib44), client will retry: rc = -110 Mar 16 03:20:30 arkle7 kernel: Lustre: Skipped 31 previous similar messages Mar 16 03:20:30 john30 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:20:30 john30 kernel: Lustre: dagg-OST000c-osc-ffff88189ac92000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:20:30 john30 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:20:30 john30 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594699400646576 length 1048576 too big: 1048516 left, 1048516 allowed Mar 16 03:20:30 john30 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:20:30 john20 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130786/real 1521130786] req@ffff8828a2b52100 x1594448327525968/t0(0) o4->dagg-OST000d-osc-ffff8817da6ad800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130830 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:30 john20 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:20:30 john20 kernel: Lustre: dagg-OST000d-osc-ffff8817da6ad800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:30 john27 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130786/real 1521130786] req@ffff881b4e2d3600 x1594448285369712/t0(0) o4->dagg-OST000f-osc-ffff88015a1e4800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130830 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:30 john27 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:20:30 john27 kernel: Lustre: dagg-OST000f-osc-ffff88015a1e4800: Connection to dagg-OST000f (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:30 john20 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:30 john20 kernel: Lustre: dagg-OST000d-osc-ffff8817da6ad800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:20:30 john27 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:30 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88290014b000 Mar 16 03:20:30 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88290014b000 Mar 16 03:20:30 john27 kernel: Lustre: dagg-OST000f-osc-ffff88015a1e4800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:20:30 john20 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:30 john27 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:30 john20 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448327737984 length 1048576 too big: 1045364 left, 1045364 allowed Mar 16 03:20:30 john20 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:20:30 john27 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448285551280 length 1048576 too big: 1046096 left, 1046096 allowed Mar 16 03:20:30 john27 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:20:30 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88290014b000 Mar 16 03:20:30 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88290014b000 Mar 16 03:20:30 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e832b6000 Mar 16 03:20:30 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e832b6000 Mar 16 03:20:31 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882baed25a00 Mar 16 03:20:31 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882baed25a00 Mar 16 03:20:31 john15 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448324100768 length 1048576 too big: 1045944 left, 1045944 allowed Mar 16 03:20:31 john15 kernel: LNetError: 906:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:20:31 john22 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130787/real 1521130787] req@ffff8823b403b900 x1594448280685024/t0(0) o4->dagg-OST000c-osc-ffff88189a897000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130831 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:31 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88290014b000 Mar 16 03:20:31 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88290014b000 Mar 16 03:20:31 john22 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:20:31 john22 kernel: Lustre: dagg-OST000c-osc-ffff88189a897000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:31 john22 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:31 john22 kernel: Lustre: dagg-OST000c-osc-ffff88189a897000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:20:31 john22 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:31 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507707400 Mar 16 03:20:31 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507707400 Mar 16 03:20:31 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8aa00 Mar 16 03:20:31 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8aa00 Mar 16 03:20:31 john21 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130787/real 1521130787] req@ffff881e6e212d00 x1594448332911728/t0(0) o4->dagg-OST000f-osc-ffff882fc5588800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130831 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:31 john21 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:20:31 john21 kernel: Lustre: dagg-OST000f-osc-ffff882fc5588800: Connection to dagg-OST000f (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:31 john21 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:31 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9767c00 Mar 16 03:20:31 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9767c00 Mar 16 03:20:31 john24 kernel: Lustre: 951:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130787/real 1521130787] req@ffff882c67f3f500 x1594448284013248/t0(0) o4->dagg-OST000c-osc-ffff882fcb645000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130831 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:31 john21 kernel: Lustre: dagg-OST000f-osc-ffff882fc5588800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:20:31 john21 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:31 john24 kernel: Lustre: 951:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:20:31 john24 kernel: Lustre: dagg-OST000c-osc-ffff882fcb645000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:31 john24 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:31 john24 kernel: Lustre: dagg-OST000c-osc-ffff882fcb645000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:20:31 john24 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:31 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507707400 Mar 16 03:20:31 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507707400 Mar 16 03:20:31 john21 kernel: LNetError: 897:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448333077632 length 1048576 too big: 1044760 left, 1044760 allowed Mar 16 03:20:31 john21 kernel: LNetError: 897:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:20:31 john19 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130787/real 1521130787] req@ffff882da4f1f800 x1594448327431632/t0(0) o4->dagg-OST000a-osc-ffff88189a323000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130831 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:20:31 john19 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:20:31 john19 kernel: Lustre: dagg-OST000a-osc-ffff88189a323000: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:31 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e832b6000 Mar 16 03:20:31 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e832b6000 Mar 16 03:20:31 john19 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:20:31 john24 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448284196816 length 1048576 too big: 1046748 left, 1046748 allowed Mar 16 03:20:31 john24 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:20:31 arkle6 kernel: Lustre: dagg-OST000a: Client d4acd478-18fd-71d3-c177-089c607f8639 (at 192.168.44.119@o2ib44) reconnecting Mar 16 03:20:31 arkle6 kernel: Lustre: Skipped 14 previous similar messages Mar 16 03:20:31 arkle6 kernel: Lustre: dagg-OST000a: Connection restored to e1a18b29-346e-ff08-5424-24003125f711 (at 192.168.44.119@o2ib44) Mar 16 03:20:31 arkle6 kernel: Lustre: Skipped 14 previous similar messages Mar 16 03:20:31 john19 kernel: Lustre: dagg-OST000a-osc-ffff88189a323000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:20:31 john19 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:20:31 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8828303d9c00 Mar 16 03:20:31 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8828303d9c00 Mar 16 03:20:31 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8827a1379200 Mar 16 03:20:31 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8827a1379200 Mar 16 03:20:31 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882b144a8600 Mar 16 03:20:31 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881646e2ba00 Mar 16 03:20:31 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880941096a00 Mar 16 03:20:31 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880941096a00 Mar 16 03:20:31 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882b144a8600 Mar 16 03:20:31 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881646e2ba00 Mar 16 03:20:31 john16 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448321675840 length 1048576 too big: 1048292 left, 1048292 allowed Mar 16 03:20:31 john16 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 9 previous similar messages Mar 16 03:20:31 arkle3 kernel: Lustre: dagg-OST0004: Client faed83b4-3a5e-2a8e-2882-7e99d174dfb0 (at 192.168.44.130@o2ib44) reconnecting Mar 16 03:20:31 arkle3 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:20:31 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8aa00 Mar 16 03:20:31 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e094e00 Mar 16 03:20:31 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0ac00 Mar 16 03:20:31 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0ac00 Mar 16 03:20:31 arkle1 kernel: LustreError: 394223:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff880ae0d54c50 x1594448284017648/t0(0) o4->6ec8b14c-8574-1b1c-979a-d89152f9126c@192.168.44.124@o2ib44:664/0 lens 608/448 e 0 to 0 dl 1521130874 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:20:31 arkle3 kernel: Lustre: dagg-OST0004: Connection restored to 2beb6461-856d-b027-2da8-040c1c990221 (at 192.168.44.130@o2ib44) Mar 16 03:20:31 arkle3 kernel: Lustre: Skipped 13 previous similar messages Mar 16 03:20:31 john26 kernel: LNetError: 904:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.35@o2ib44, match 1594448281497904 length 1048576 too big: 1045572 left, 1045572 allowed Mar 16 03:20:31 john26 kernel: LNetError: 904:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:20:31 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c6c9af200 Mar 16 03:20:31 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c6c9af200 Mar 16 03:20:31 arkle1 kernel: LustreError: 394223:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 29 previous similar messages Mar 16 03:20:31 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808f18c9a00 Mar 16 03:20:31 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808f18c9a00 Mar 16 03:20:31 arkle1 kernel: Lustre: dagg-OST0000: Bulk IO write error with 219d9f55-42f0-6656-ab6f-f4de9ef8c6c7 (at 192.168.44.127@o2ib44), client will retry: rc = -110 Mar 16 03:20:31 arkle1 kernel: Lustre: Skipped 29 previous similar messages Mar 16 03:20:31 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88169cc58000 Mar 16 03:20:31 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88169cc58000 Mar 16 03:20:31 arkle5 kernel: LustreError: 449866:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881619b02050 x1594448281318416/t0(0) o4->f4a3cf53-7286-0d27-5cb4-5f6321872642@192.168.44.126@o2ib44:664/0 lens 608/448 e 0 to 0 dl 1521130874 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:20:31 arkle5 kernel: LustreError: 449866:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 31 previous similar messages Mar 16 03:20:31 john18 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130787/real 1521130787] req@ffff882b86330900 x1594448284367264/t0(0) o4->dagg-OST0000-osc-ffff88015a243000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130831 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:31 john18 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:20:31 john18 kernel: Lustre: dagg-OST0000-osc-ffff88015a243000: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:31 arkle5 kernel: Lustre: dagg-OST0008: Bulk IO write error with f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44), client will retry: rc = -110 Mar 16 03:20:31 arkle5 kernel: Lustre: Skipped 31 previous similar messages Mar 16 03:20:31 john18 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:20:31 john18 kernel: Lustre: dagg-OST0000-osc-ffff88015a243000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:20:31 john18 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:20:31 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a422400 Mar 16 03:20:31 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a422400 Mar 16 03:20:31 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac950c00 Mar 16 03:20:31 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac950c00 Mar 16 03:20:31 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dc52f0400 Mar 16 03:20:31 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dc52f0400 Mar 16 03:20:31 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8aa00 Mar 16 03:20:31 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e094e00 Mar 16 03:20:31 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882baed24200 Mar 16 03:20:31 arkle4 kernel: LustreError: 294183:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882baed24200 Mar 16 03:20:31 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a422400 Mar 16 03:20:31 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a422400 Mar 16 03:20:31 john8 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130787/real 1521130787] req@ffff88270812b300 x1594448316803760/t0(0) o4->dagg-OST0000-osc-ffff88015a2fe000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130831 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:31 john8 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:20:31 john8 kernel: Lustre: dagg-OST0000-osc-ffff88015a2fe000: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:31 john8 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:20:31 john8 kernel: Lustre: dagg-OST0000-osc-ffff88015a2fe000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:20:31 john8 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:20:31 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f18757a00 Mar 16 03:20:31 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f18757a00 Mar 16 03:20:31 john8 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448316970064 length 1048576 too big: 1047284 left, 1047284 allowed Mar 16 03:20:31 john8 kernel: LNetError: 916:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:20:31 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e7ac19800 Mar 16 03:20:31 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817583a2c00 Mar 16 03:20:31 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e7ac19800 Mar 16 03:20:31 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0ac00 Mar 16 03:20:31 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0ac00 Mar 16 03:20:31 john33 kernel: Lustre: 940:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130787/real 1521130787] req@ffff882e3f367500 x1594699393878992/t0(0) o4->dagg-OST000a-osc-ffff88006bb84800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130831 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:31 john33 kernel: Lustre: 940:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:20:31 john33 kernel: Lustre: dagg-OST000a-osc-ffff88006bb84800: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:31 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817583a2c00 Mar 16 03:20:31 john33 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:31 john13 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130787/real 1521130787] req@ffff880895262d00 x1594448319196096/t0(0) o4->dagg-OST0009-osc-ffff88015a2c6800@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130831 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:31 john13 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:20:31 john13 kernel: Lustre: dagg-OST0007-osc-ffff88015a2c6800: Connection to dagg-OST0007 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:31 john13 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:20:31 john13 kernel: Lustre: dagg-OST0007-osc-ffff88015a2c6800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:20:31 john13 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:20:31 john13 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448319377888 length 1048576 too big: 1048108 left, 1048108 allowed Mar 16 03:20:31 john13 kernel: LNetError: 910:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:20:31 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dc52f0400 Mar 16 03:20:31 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dc52f0400 Mar 16 03:20:31 john33 kernel: Lustre: dagg-OST000a-osc-ffff88006bb84800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:20:31 john33 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:31 john32 kernel: Lustre: 928:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130787/real 1521130787] req@ffff882182fbef00 x1594699396464000/t0(0) o4->dagg-OST000d-osc-ffff8817c5384000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130831 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:31 john32 kernel: Lustre: 928:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:20:31 john32 kernel: Lustre: dagg-OST000d-osc-ffff8817c5384000: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:31 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8aa00 Mar 16 03:20:31 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8aa00 Mar 16 03:20:31 john9 kernel: Lustre: 935:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130787/real 1521130787] req@ffff88258cc91b00 x1594448323945264/t0(0) o4->dagg-OST0004-osc-ffff882fdd33b800@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130831 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:31 john9 kernel: Lustre: 935:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:20:31 john9 kernel: Lustre: dagg-OST0004-osc-ffff882fdd33b800: Connection to dagg-OST0004 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:31 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881717626200 Mar 16 03:20:31 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881717626200 Mar 16 03:20:31 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880598b84800 Mar 16 03:20:31 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880598b84800 Mar 16 03:20:31 john32 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:20:31 john33 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594699394045072 length 1048576 too big: 1046348 left, 1046348 allowed Mar 16 03:20:31 john33 kernel: LNetError: 900:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:20:31 john9 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:31 john32 kernel: Lustre: dagg-OST000d-osc-ffff8817c5384000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:20:31 john32 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:20:31 john9 kernel: Lustre: dagg-OST0004-osc-ffff882fdd33b800: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:20:31 john9 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:31 john32 kernel: LNetError: 897:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594699396627920 length 1048576 too big: 1048540 left, 1048540 allowed Mar 16 03:20:31 john32 kernel: LNetError: 897:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:20:31 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8827a1379200 Mar 16 03:20:31 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f18757a00 Mar 16 03:20:31 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f18757a00 Mar 16 03:20:31 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8827a1379200 Mar 16 03:20:31 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880b248a3c00 Mar 16 03:20:31 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880b248a3c00 Mar 16 03:20:31 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f2aaa9e00 Mar 16 03:20:31 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f2aaa9e00 Mar 16 03:20:31 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f46287200 Mar 16 03:20:31 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881717627e00 Mar 16 03:20:31 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881717627e00 Mar 16 03:20:31 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f46287200 Mar 16 03:20:31 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507703600 Mar 16 03:20:31 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507703600 Mar 16 03:20:31 john29 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130787/real 1521130787] req@ffff882c3e222d00 x1594699419066272/t0(0) o4->dagg-OST0005-osc-ffff88189afee000@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130831 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:20:31 john29 kernel: Lustre: dagg-OST0005-osc-ffff88189afee000: Connection to dagg-OST0005 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:31 john29 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:20:31 john29 kernel: Lustre: dagg-OST0005-osc-ffff88189afee000: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:20:31 john29 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:20:31 john29 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594699419228432 length 1048576 too big: 1046768 left, 1046768 allowed Mar 16 03:20:31 john29 kernel: LNetError: 902:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:20:31 john29 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 16 03:20:31 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881717627e00 Mar 16 03:20:31 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881717627e00 Mar 16 03:20:31 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f2aaa9e00 Mar 16 03:20:31 arkle5 kernel: LustreError: 294089:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f2aaa9e00 Mar 16 03:20:32 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c51847600 Mar 16 03:20:32 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c51847600 Mar 16 03:20:32 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880598b84800 Mar 16 03:20:32 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88150d7eb400 Mar 16 03:20:32 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9764800 Mar 16 03:20:32 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9764800 Mar 16 03:20:32 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880598b84800 Mar 16 03:20:32 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88150d7eb400 Mar 16 03:20:32 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880598b84800 Mar 16 03:20:32 arkle2 kernel: LustreError: 294378:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880598b84800 Mar 16 03:20:32 john6 kernel: Lustre: 954:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130788/real 1521130788] req@ffff8819dca5c200 x1594448323927280/t0(0) o4->dagg-OST0000-osc-ffff881899c73800@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130832 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:32 john6 kernel: Lustre: dagg-OST0002-osc-ffff881899c73800: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:32 john6 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:20:32 john6 kernel: Lustre: dagg-OST0002-osc-ffff881899c73800: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:20:32 john6 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:20:32 john6 kernel: Lustre: 954:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 16 03:20:32 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0e800 Mar 16 03:20:32 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0e800 Mar 16 03:20:32 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a426200 Mar 16 03:20:32 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a426200 Mar 16 03:20:32 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88150d7ef000 Mar 16 03:20:32 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8815a8fc1400 Mar 16 03:20:32 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894afc00 Mar 16 03:20:32 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f46287200 Mar 16 03:20:32 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88150d7ef000 Mar 16 03:20:32 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894afc00 Mar 16 03:20:32 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8815a8fc1400 Mar 16 03:20:32 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f46287200 Mar 16 03:20:32 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9764800 Mar 16 03:20:32 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9764800 Mar 16 03:20:32 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0e800 Mar 16 03:20:32 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0e800 Mar 16 03:20:32 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f2aaa9e00 Mar 16 03:20:32 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f2aaa9e00 Mar 16 03:20:32 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9764800 Mar 16 03:20:32 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9764800 Mar 16 03:20:32 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0e800 Mar 16 03:20:32 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0e800 Mar 16 03:20:32 john31 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594699394332288 length 1048576 too big: 1048180 left, 1048180 allowed Mar 16 03:20:32 john31 kernel: LNetError: 896:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:20:32 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9764800 Mar 16 03:20:32 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9764800 Mar 16 03:20:32 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8827a1379200 Mar 16 03:20:32 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8827a1379200 Mar 16 03:20:32 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac950c00 Mar 16 03:20:32 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac950c00 Mar 16 03:20:32 john7 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130788/real 1521130788] req@ffff881951f74500 x1594448319405744/t0(0) o4->dagg-OST0001-osc-ffff882fda4b4000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130832 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:32 john7 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:20:32 john7 kernel: Lustre: dagg-OST0001-osc-ffff882fda4b4000: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:32 john7 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:20:32 john7 kernel: Lustre: dagg-OST0001-osc-ffff882fda4b4000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:20:32 john7 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:20:32 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac950c00 Mar 16 03:20:32 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac950c00 Mar 16 03:20:32 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f2aaa9e00 Mar 16 03:20:32 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f2aaa9e00 Mar 16 03:20:32 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9764800 Mar 16 03:20:32 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9764800 Mar 16 03:20:32 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a420e00 Mar 16 03:20:32 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a420e00 Mar 16 03:20:32 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882baed22c00 Mar 16 03:20:32 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882baed22c00 Mar 16 03:20:32 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8b800 Mar 16 03:20:32 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8b800 Mar 16 03:20:32 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac950c00 Mar 16 03:20:32 arkle1 kernel: LustreError: 294169:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac950c00 Mar 16 03:20:32 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c51847600 Mar 16 03:20:32 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c51847600 Mar 16 03:20:32 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882baed25400 Mar 16 03:20:32 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882baed25400 Mar 16 03:20:32 john23 kernel: Lustre: 924:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130788/real 1521130788] req@ffff8814da225d00 x1594448282171968/t0(0) o4->dagg-OST0002-osc-ffff882fdbc22800@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130832 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:32 john23 kernel: Lustre: dagg-OST0001-osc-ffff882fdbc22800: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:32 john23 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:20:32 john23 kernel: Lustre: dagg-OST0001-osc-ffff882fdbc22800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:20:32 john23 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:20:32 john23 kernel: Lustre: 924:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:20:32 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9764e00 Mar 16 03:20:32 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9764e00 Mar 16 03:20:33 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9764e00 Mar 16 03:20:33 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9764e00 Mar 16 03:20:33 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88290014b000 Mar 16 03:20:33 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88290014b000 Mar 16 03:20:33 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507701800 Mar 16 03:20:33 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507701800 Mar 16 03:20:33 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f46287200 Mar 16 03:20:33 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f46287200 Mar 16 03:20:33 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a426800 Mar 16 03:20:33 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a426800 Mar 16 03:20:33 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8827a1379600 Mar 16 03:20:33 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8827a1379600 Mar 16 03:20:33 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8b800 Mar 16 03:20:33 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8b800 Mar 16 03:20:33 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88015a425400 Mar 16 03:20:33 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88015a425400 Mar 16 03:20:33 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880941094c00 Mar 16 03:20:33 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880941094c00 Mar 16 03:20:33 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882bcb055600 Mar 16 03:20:33 arkle5 kernel: LustreError: 294094:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882bcb055600 Mar 16 03:20:34 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882dac950c00 Mar 16 03:20:34 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882dac950c00 Mar 16 03:20:34 john10 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130790/real 1521130790] req@ffff881757b33300 x1594448319886272/t0(0) o4->dagg-OST000f-osc-ffff882fd4b83800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130834 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:34 john10 kernel: Lustre: dagg-OST0001-osc-ffff882fd4b83800: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:34 john10 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:34 john10 kernel: Lustre: dagg-OST0001-osc-ffff882fd4b83800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:20:34 john10 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:20:34 john10 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448320049824 length 1048576 too big: 1046492 left, 1046492 allowed Mar 16 03:20:34 john10 kernel: LNetError: 915:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:20:34 john10 kernel: Lustre: 932:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:20:34 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88058bf26400 Mar 16 03:20:34 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88058bf26400 Mar 16 03:20:34 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881717625a00 Mar 16 03:20:34 arkle3 kernel: LustreError: 294215:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881717625a00 Mar 16 03:20:34 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0e800 Mar 16 03:20:34 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0e800 Mar 16 03:20:34 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8b800 Mar 16 03:20:34 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8b800 Mar 16 03:20:34 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88005a83e000 Mar 16 03:20:34 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88005a83e000 Mar 16 03:20:34 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8827a1379400 Mar 16 03:20:34 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8827a1379400 Mar 16 03:20:34 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0a000 Mar 16 03:20:34 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0a000 Mar 16 03:20:34 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882baed25800 Mar 16 03:20:34 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882baed25800 Mar 16 03:20:34 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177f58de00 Mar 16 03:20:34 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177f58de00 Mar 16 03:20:35 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177f58de00 Mar 16 03:20:35 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177f58de00 Mar 16 03:20:35 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817cea2e200 Mar 16 03:20:35 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817cea2e200 Mar 16 03:20:35 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8827a137f800 Mar 16 03:20:35 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8827a137f800 Mar 16 03:20:36 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f2eb25600 Mar 16 03:20:36 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f2eb25600 Mar 16 03:20:36 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ee12a6000 Mar 16 03:20:36 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ee12a6000 Mar 16 03:20:36 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507706a00 Mar 16 03:20:36 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507706a00 Mar 16 03:20:36 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9763200 Mar 16 03:20:36 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9763200 Mar 16 03:20:37 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f2eb27400 Mar 16 03:20:37 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f2eb27400 Mar 16 03:20:37 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8815f1877e00 Mar 16 03:20:37 arkle7 kernel: LustreError: 294144:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8815f1877e00 Mar 16 03:20:38 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88271f44d000 Mar 16 03:20:38 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88271f44d000 Mar 16 03:20:38 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88271f44d000 Mar 16 03:20:38 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88271f44d000 Mar 16 03:20:38 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882089acfc00 Mar 16 03:20:38 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882089acfc00 Mar 16 03:20:39 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f46282000 Mar 16 03:20:39 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f46282000 Mar 16 03:20:39 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9764600 Mar 16 03:20:39 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9764600 Mar 16 03:20:39 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8815f1876800 Mar 16 03:20:39 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8815f1876800 Mar 16 03:20:40 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f09800 Mar 16 03:20:40 arkle1 kernel: LustreError: 294164:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f09800 Mar 16 03:20:40 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f46281800 Mar 16 03:20:40 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f46281800 Mar 16 03:20:42 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817cea29c00 Mar 16 03:20:42 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817cea29c00 Mar 16 03:20:43 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507703800 Mar 16 03:20:43 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507703800 Mar 16 03:20:51 arkle8 kernel: Lustre: dagg-OST000e: Connection restored to d5de0e2d-4431-41fa-738a-2fa7d0a3c365 (at 192.168.44.126@o2ib44) Mar 16 03:20:51 arkle8 kernel: Lustre: Skipped 19 previous similar messages Mar 16 03:20:51 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177f58f600 Mar 16 03:20:51 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177f58f600 Mar 16 03:20:51 john11 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130807/real 1521130807] req@ffff8826ea14ad00 x1594448320168128/t0(0) o4->dagg-OST000a-osc-ffff882fd571f000@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130851 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:51 john11 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:20:51 john11 kernel: Lustre: dagg-OST000a-osc-ffff882fd571f000: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:51 john11 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:20:51 john11 kernel: Lustre: dagg-OST000a-osc-ffff882fd571f000: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:20:51 john11 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:20:51 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8813e3fe5c00 Mar 16 03:20:51 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8813e3fe5c00 Mar 16 03:20:51 john25 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130807/real 1521130807] req@ffff8824baa20f00 x1594448280895840/t0(0) o4->dagg-OST000e-osc-ffff882fd7db5800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130851 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:51 john25 kernel: Lustre: 946:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Mar 16 03:20:51 john25 kernel: Lustre: dagg-OST000e-osc-ffff882fd7db5800: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:51 john25 kernel: Lustre: Skipped 8 previous similar messages Mar 16 03:20:52 john25 kernel: Lustre: dagg-OST000e-osc-ffff882fd7db5800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:20:52 john25 kernel: Lustre: Skipped 8 previous similar messages Mar 16 03:20:52 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8824cbb5f200 Mar 16 03:20:52 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8824cbb5f200 Mar 16 03:20:52 arkle8 kernel: LustreError: 298958:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8824cbb5f200 Mar 16 03:20:52 arkle8 kernel: LustreError: 298958:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8824cbb5f200 Mar 16 03:20:52 arkle8 kernel: LustreError: 298958:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808ac2ab000 Mar 16 03:20:52 arkle8 kernel: LustreError: 298958:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808ac2ab000 Mar 16 03:20:53 john15 kernel: Lustre: 919:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130809/real 1521130809] req@ffff880e86ebb300 x1594448323943024/t0(0) o4->dagg-OST000e-osc-ffff882fde9f4800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130853 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:20:53 john15 kernel: Lustre: 919:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Mar 16 03:20:53 john15 kernel: Lustre: dagg-OST000e-osc-ffff882fde9f4800: Connection to dagg-OST000e (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:20:53 john15 kernel: Lustre: Skipped 6 previous similar messages Mar 16 03:20:53 john15 kernel: Lustre: dagg-OST000e-osc-ffff882fde9f4800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:20:53 john15 kernel: Lustre: Skipped 6 previous similar messages Mar 16 03:20:53 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808ac2ab000 Mar 16 03:20:53 arkle8 kernel: LustreError: 298959:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808ac2ab000 Mar 16 03:20:53 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177f58fc00 Mar 16 03:20:53 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88178ce14c00 Mar 16 03:20:53 arkle8 kernel: LustreError: 298953:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177f58fc00 Mar 16 03:20:53 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88178ce14c00 Mar 16 03:20:54 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8829271be200 Mar 16 03:20:54 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8829271be200 Mar 16 03:21:14 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882baed27200 Mar 16 03:21:14 arkle4 kernel: LustreError: 294182:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882baed27200 Mar 16 03:21:14 arkle4 kernel: LustreError: 450256:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff882d66e7c850 x1594448280872016/t0(0) o4->40d9b451-fab3-be40-89fb-0c9b04087671@192.168.44.125@o2ib44:707/0 lens 608/448 e 0 to 0 dl 1521130917 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:21:14 arkle4 kernel: LustreError: 450256:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 29 previous similar messages Mar 16 03:21:14 arkle4 kernel: Lustre: dagg-OST0006: Bulk IO write error with 40d9b451-fab3-be40-89fb-0c9b04087671 (at 192.168.44.125@o2ib44), client will retry: rc = -110 Mar 16 03:21:14 arkle4 kernel: Lustre: Skipped 31 previous similar messages Mar 16 03:21:14 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8b000 Mar 16 03:21:14 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8b000 Mar 16 03:21:14 john14 kernel: Lustre: 937:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130830/real 1521130830] req@ffff882dddbaef00 x1594448317079152/t0(0) o4->dagg-OST0006-osc-ffff88189a368800@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130874 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:21:14 john14 kernel: Lustre: dagg-OST0006-osc-ffff88189a368800: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:14 john14 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:21:14 john14 kernel: Lustre: dagg-OST0006-osc-ffff88189a368800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:21:14 john14 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:21:14 john14 kernel: Lustre: 937:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:21:14 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8b000 Mar 16 03:21:14 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8b000 Mar 16 03:21:14 arkle7 kernel: Lustre: dagg-OST000c: Client 23017e25-beb4-2d0c-1e7b-9a1bd7cbb415 (at 192.168.44.114@o2ib44) reconnecting Mar 16 03:21:14 arkle7 kernel: Lustre: Skipped 31 previous similar messages Mar 16 03:21:14 arkle7 kernel: Lustre: dagg-OST000c: Connection restored to 78033fff-f4e3-2589-3920-d6d5c19d59a2 (at 192.168.44.114@o2ib44) Mar 16 03:21:14 arkle7 kernel: Lustre: Skipped 31 previous similar messages Mar 16 03:21:14 john28 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130830/real 1521130830] req@ffff881aaab92d00 x1594448282246752/t0(0) o4->dagg-OST0006-osc-ffff8817d9cf4000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130874 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:14 john28 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Mar 16 03:21:14 john28 kernel: Lustre: dagg-OST0006-osc-ffff8817d9cf4000: Connection to dagg-OST0006 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:14 john28 kernel: Lustre: Skipped 9 previous similar messages Mar 16 03:21:14 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881507700000 Mar 16 03:21:14 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881507700000 Mar 16 03:21:14 john28 kernel: Lustre: dagg-OST0006-osc-ffff8817d9cf4000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:21:14 john28 kernel: Lustre: Skipped 9 previous similar messages Mar 16 03:21:14 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e281e6800 Mar 16 03:21:14 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e281e6800 Mar 16 03:21:14 john30 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130830/real 1521130830] req@ffff882e4338a400 x1594699400481984/t0(0) o4->dagg-OST000c-osc-ffff88189ac92000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130874 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:14 john30 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:21:14 john30 kernel: Lustre: dagg-OST000c-osc-ffff88189ac92000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:14 john30 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:21:14 john30 kernel: Lustre: dagg-OST000c-osc-ffff88189ac92000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:21:14 john30 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:21:14 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882a70bd4000 Mar 16 03:21:14 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882a70bd4000 Mar 16 03:21:14 john20 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130830/real 1521130830] req@ffff8828a2b52100 x1594448327525968/t0(0) o4->dagg-OST000d-osc-ffff8817da6ad800@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130874 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:14 john20 kernel: Lustre: 939:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:21:14 john20 kernel: Lustre: dagg-OST000d-osc-ffff8817da6ad800: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:14 john20 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:21:14 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e281e6800 Mar 16 03:21:14 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e281e6800 Mar 16 03:21:15 john20 kernel: Lustre: dagg-OST000d-osc-ffff8817da6ad800: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:21:15 john27 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130830/real 1521130830] req@ffff881b4e2d3600 x1594448285369712/t0(0) o4->dagg-OST000f-osc-ffff88015a1e4800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130874 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:15 john27 kernel: Lustre: 947:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:21:15 john27 kernel: Lustre: dagg-OST000f-osc-ffff88015a1e4800: Connection to dagg-OST000f (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:15 john27 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:21:15 john27 kernel: Lustre: dagg-OST000f-osc-ffff88015a1e4800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:21:15 john20 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:21:15 john27 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:21:15 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882a70bd4000 Mar 16 03:21:15 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882a70bd4000 Mar 16 03:21:15 john27 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.38@o2ib44, match 1594448285574992 length 1048576 too big: 1046096 left, 1046096 allowed Mar 16 03:21:15 john27 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 4 previous similar messages Mar 16 03:21:15 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882a70bd4000 Mar 16 03:21:15 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882a70bd4000 Mar 16 03:21:15 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88177f58fc00 Mar 16 03:21:15 arkle8 kernel: LustreError: 298955:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88177f58fc00 Mar 16 03:21:15 john22 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448280875344 length 1048576 too big: 1048436 left, 1048436 allowed Mar 16 03:21:15 john22 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 7 previous similar messages Mar 16 03:21:15 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882a70095e00 Mar 16 03:21:15 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882a70095e00 Mar 16 03:21:15 john17 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130831/real 1521130831] req@ffff880e00b31500 x1594448316853392/t0(0) o4->dagg-OST0002-osc-ffff881899e5b000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130875 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:15 john17 kernel: Lustre: 927:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Mar 16 03:21:15 john17 kernel: Lustre: dagg-OST0002-osc-ffff881899e5b000: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:15 john17 kernel: Lustre: Skipped 7 previous similar messages Mar 16 03:21:15 arkle2 kernel: Lustre: dagg-OST0002: Client 6b4209e2-620e-030d-e2d1-b9c575162077 (at 192.168.44.117@o2ib44) reconnecting Mar 16 03:21:15 arkle2 kernel: Lustre: Skipped 27 previous similar messages Mar 16 03:21:15 john21 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130831/real 1521130831] req@ffff881e6e212d00 x1594448332911728/t0(0) o4->dagg-OST000f-osc-ffff882fc5588800@192.168.44.38@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130875 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:15 john24 kernel: Lustre: 951:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130831/real 1521130831] req@ffff882c67f3f500 x1594448284013248/t0(0) o4->dagg-OST000c-osc-ffff882fcb645000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130875 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:15 john24 kernel: Lustre: 951:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:21:15 john24 kernel: Lustre: dagg-OST000c-osc-ffff882fcb645000: Connection to dagg-OST000c (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:15 arkle2 kernel: Lustre: dagg-OST0002: Connection restored to 7002ccab-a112-d1d5-6fdf-c8c42ad342fc (at 192.168.44.117@o2ib44) Mar 16 03:21:15 arkle2 kernel: Lustre: Skipped 27 previous similar messages Mar 16 03:21:15 john24 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:21:15 john21 kernel: Lustre: 926:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:21:15 john21 kernel: Lustre: dagg-OST000f-osc-ffff882fc5588800: Connection to dagg-OST000f (at 192.168.44.38@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:15 john21 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:21:15 john24 kernel: Lustre: dagg-OST000c-osc-ffff882fcb645000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:21:15 john21 kernel: Lustre: dagg-OST000f-osc-ffff882fc5588800: Connection restored to 192.168.44.38@o2ib44 (at 192.168.44.38@o2ib44) Mar 16 03:21:15 john17 kernel: Lustre: dagg-OST0002-osc-ffff881899e5b000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:21:15 john17 kernel: Lustre: Skipped 7 previous similar messages Mar 16 03:21:15 john21 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:21:15 john24 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:21:15 john19 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.36@o2ib44, match 1594448327644416 length 1048576 too big: 1045432 left, 1045432 allowed Mar 16 03:21:15 john19 kernel: LNetError: 899:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 7 previous similar messages Mar 16 03:21:15 john17 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448317063280 length 1048576 too big: 1044508 left, 1044508 allowed Mar 16 03:21:15 john17 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 7 previous similar messages Mar 16 03:21:15 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8811eee43e00 Mar 16 03:21:15 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8811eee43e00 Mar 16 03:21:15 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f18752e00 Mar 16 03:21:15 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8813e3fe7c00 Mar 16 03:21:15 john24 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.37@o2ib44, match 1594448284220928 length 1048576 too big: 1046748 left, 1046748 allowed Mar 16 03:21:15 john24 kernel: LNetError: 905:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:21:15 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f18752e00 Mar 16 03:21:15 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881ce9766000 Mar 16 03:21:15 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881ce9766000 Mar 16 03:21:15 arkle2 kernel: LustreError: 350069:0:(ldlm_lib.c:3242:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88219a97d850 x1594448316853392/t0(0) o4->6b4209e2-620e-030d-e2d1-b9c575162077@192.168.44.117@o2ib44:708/0 lens 608/448 e 0 to 0 dl 1521130918 ref 1 fl Interpret:/2/0 rc 0/0 Mar 16 03:21:15 arkle6 kernel: LustreError: 294096:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8813e3fe7c00 Mar 16 03:21:15 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882a70095e00 Mar 16 03:21:15 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882a70095e00 Mar 16 03:21:15 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817947be600 Mar 16 03:21:15 arkle2 kernel: LustreError: 350069:0:(ldlm_lib.c:3242:target_bulk_io()) Skipped 31 previous similar messages Mar 16 03:21:15 john12 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130831/real 1521130831] req@ffff8809853abc00 x1594448324464560/t0(0) o4->dagg-OST000a-osc-ffff88015a3ab800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130875 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:15 john12 kernel: Lustre: 930:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:21:15 john12 kernel: Lustre: dagg-OST000a-osc-ffff88015a3ab800: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:15 arkle6 kernel: LustreError: 294098:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817947be600 Mar 16 03:21:15 arkle2 kernel: Lustre: dagg-OST0002: Bulk IO write error with 6b4209e2-620e-030d-e2d1-b9c575162077 (at 192.168.44.117@o2ib44), client will retry: rc = -110 Mar 16 03:21:15 arkle2 kernel: Lustre: Skipped 31 previous similar messages Mar 16 03:21:15 john12 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:21:15 john12 kernel: Lustre: dagg-OST000a-osc-ffff88015a3ab800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:21:15 john12 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:21:15 arkle1 kernel: Lustre: dagg-OST0000: Client 6ec8b14c-8574-1b1c-979a-d89152f9126c (at 192.168.44.124@o2ib44) reconnecting Mar 16 03:21:15 arkle1 kernel: Lustre: Skipped 29 previous similar messages Mar 16 03:21:15 arkle1 kernel: Lustre: dagg-OST0000: Connection restored to 8f94461e-6f3c-b18c-f4e9-0d4fa2a51df8 (at 192.168.44.124@o2ib44) Mar 16 03:21:15 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882089acd200 Mar 16 03:21:15 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882089acd200 Mar 16 03:21:15 arkle1 kernel: Lustre: Skipped 29 previous similar messages Mar 16 03:21:15 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8b000 Mar 16 03:21:15 arkle4 kernel: LustreError: 294179:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8b000 Mar 16 03:21:15 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c6c9ade00 Mar 16 03:21:15 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c6c9ade00 Mar 16 03:21:15 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0f600 Mar 16 03:21:15 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0f600 Mar 16 03:21:15 john26 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130831/real 1521130831] req@ffff882a53e52700 x1594448281318416/t0(0) o4->dagg-OST0008-osc-ffff882fde832000@192.168.44.35@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130875 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:15 john26 kernel: Lustre: 944:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Mar 16 03:21:15 john26 kernel: Lustre: dagg-OST0008-osc-ffff882fde832000: Connection to dagg-OST0008 (at 192.168.44.35@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:15 john16 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130831/real 1521130831] req@ffff881e67300f00 x1594448321487984/t0(0) o4->dagg-OST0007-osc-ffff882fc87e8000@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130875 ref 2 fl Rpc:X/2/ffffffff rc -11/-1 Mar 16 03:21:15 john16 kernel: Lustre: dagg-OST0007-osc-ffff882fc87e8000: Connection to dagg-OST0007 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:15 john16 kernel: Lustre: Skipped 7 previous similar messages Mar 16 03:21:15 john16 kernel: Lustre: dagg-OST0007-osc-ffff882fc87e8000: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:21:15 john16 kernel: Lustre: Skipped 7 previous similar messages Mar 16 03:21:15 john16 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Mar 16 03:21:15 john26 kernel: Lustre: Skipped 7 previous similar messages Mar 16 03:21:15 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8b000 Mar 16 03:21:15 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8b000 Mar 16 03:21:15 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e21a6c000 Mar 16 03:21:15 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e21a6c000 Mar 16 03:21:15 arkle5 kernel: Lustre: dagg-OST0008: Client f4a3cf53-7286-0d27-5cb4-5f6321872642 (at 192.168.44.126@o2ib44) reconnecting Mar 16 03:21:15 arkle5 kernel: Lustre: Skipped 31 previous similar messages Mar 16 03:21:15 arkle5 kernel: Lustre: dagg-OST0008: Connection restored to d5de0e2d-4431-41fa-738a-2fa7d0a3c365 (at 192.168.44.126@o2ib44) Mar 16 03:21:15 arkle5 kernel: Lustre: Skipped 31 previous similar messages Mar 16 03:21:15 john26 kernel: Lustre: dagg-OST0008-osc-ffff882fde832000: Connection restored to 192.168.44.35@o2ib44 (at 192.168.44.35@o2ib44) Mar 16 03:21:15 john26 kernel: Lustre: Skipped 7 previous similar messages Mar 16 03:21:15 john18 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448284579696 length 1048576 too big: 1048556 left, 1048556 allowed Mar 16 03:21:15 john18 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 9 previous similar messages Mar 16 03:21:15 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817cea2bc00 Mar 16 03:21:15 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882b83e1d800 Mar 16 03:21:15 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882b83e1d800 Mar 16 03:21:15 arkle5 kernel: LustreError: 294091:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817cea2bc00 Mar 16 03:21:15 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881c6c9ade00 Mar 16 03:21:15 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881c6c9ade00 Mar 16 03:21:15 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881cb001d200 Mar 16 03:21:15 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881cb001d200 Mar 16 03:21:15 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0f600 Mar 16 03:21:15 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0f600 Mar 16 03:21:15 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882b83e1d800 Mar 16 03:21:15 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882b83e1d800 Mar 16 03:21:15 arkle8 kernel: LustreError: 298958:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808ac2ab800 Mar 16 03:21:15 arkle8 kernel: LustreError: 298958:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808ac2ab800 Mar 16 03:21:15 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f18754600 Mar 16 03:21:15 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f18754600 Mar 16 03:21:15 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e7ac1dc00 Mar 16 03:21:15 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8829271ba800 Mar 16 03:21:15 john8 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130831/real 1521130831] req@ffff88270812b300 x1594448316803760/t0(0) o4->dagg-OST0000-osc-ffff88015a2fe000@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130875 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:15 john8 kernel: Lustre: 945:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:21:15 john8 kernel: Lustre: dagg-OST0000-osc-ffff88015a2fe000: Connection to dagg-OST0000 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:15 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e7ac1dc00 Mar 16 03:21:15 arkle3 kernel: LustreError: 294213:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8829271ba800 Mar 16 03:21:15 john8 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:21:15 john8 kernel: Lustre: dagg-OST0000-osc-ffff88015a2fe000: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:21:15 john8 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:21:15 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c282ffa00 Mar 16 03:21:15 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c282ffa00 Mar 16 03:21:15 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0f600 Mar 16 03:21:15 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0f600 Mar 16 03:21:15 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880df8b8b000 Mar 16 03:21:15 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880df8b8b000 Mar 16 03:21:15 john13 kernel: Lustre: 941:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130831/real 1521130831] req@ffff88221c8d0f00 x1594448319199392/t0(0) o4->dagg-OST0007-osc-ffff88015a2c6800@192.168.44.34@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130875 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:15 john13 kernel: Lustre: 941:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 03:21:15 john13 kernel: Lustre: dagg-OST0007-osc-ffff88015a2c6800: Connection to dagg-OST0007 (at 192.168.44.34@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:15 john29 kernel: Lustre: 936:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130831/real 1521130831] req@ffff882c3e225a00 x1594699419066352/t0(0) o4->dagg-OST0005-osc-ffff88189afee000@192.168.44.33@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130875 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:15 john29 kernel: Lustre: 936:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:21:15 john29 kernel: Lustre: dagg-OST0005-osc-ffff88189afee000: Connection to dagg-OST0005 (at 192.168.44.33@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:15 john33 kernel: Lustre: 940:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130831/real 1521130831] req@ffff882e3f367500 x1594699393878992/t0(0) o4->dagg-OST000a-osc-ffff88006bb84800@192.168.44.36@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130875 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:15 john33 kernel: Lustre: 940:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 16 03:21:15 john33 kernel: Lustre: dagg-OST000a-osc-ffff88006bb84800: Connection to dagg-OST000a (at 192.168.44.36@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:15 john13 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:21:15 john29 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:21:15 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882f18754600 Mar 16 03:21:15 arkle6 kernel: LustreError: 294097:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882f18754600 Mar 16 03:21:15 john33 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:21:15 john13 kernel: Lustre: dagg-OST0007-osc-ffff88015a2c6800: Connection restored to 192.168.44.34@o2ib44 (at 192.168.44.34@o2ib44) Mar 16 03:21:15 john13 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:21:15 john29 kernel: Lustre: dagg-OST0005-osc-ffff88189afee000: Connection restored to 192.168.44.33@o2ib44 (at 192.168.44.33@o2ib44) Mar 16 03:21:15 john29 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594699419252368 length 1048576 too big: 1045736 left, 1045736 allowed Mar 16 03:21:15 john29 kernel: LNetError: 901:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 3 previous similar messages Mar 16 03:21:15 john33 kernel: Lustre: dagg-OST000a-osc-ffff88006bb84800: Connection restored to 192.168.44.36@o2ib44 (at 192.168.44.36@o2ib44) Mar 16 03:21:15 john33 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:21:15 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880e7ac1dc00 Mar 16 03:21:15 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880e7ac1dc00 Mar 16 03:21:15 john32 kernel: Lustre: 928:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130831/real 1521130831] req@ffff882182fbef00 x1594699396464000/t0(0) o4->dagg-OST000d-osc-ffff8817c5384000@192.168.44.37@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130875 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:15 john32 kernel: Lustre: 928:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:21:15 john32 kernel: Lustre: dagg-OST000d-osc-ffff8817c5384000: Connection to dagg-OST000d (at 192.168.44.37@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:15 john9 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.33@o2ib44, match 1594448324162384 length 1048576 too big: 1044696 left, 1044696 allowed Mar 16 03:21:15 john9 kernel: LNetError: 907:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 9 previous similar messages Mar 16 03:21:15 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8829271ba800 Mar 16 03:21:15 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c282ffa00 Mar 16 03:21:15 arkle3 kernel: LustreError: 294220:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c282ffa00 Mar 16 03:21:15 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882a70091a00 Mar 16 03:21:15 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e094e00 Mar 16 03:21:15 arkle4 kernel: LustreError: 294177:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e094e00 Mar 16 03:21:15 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882a70091a00 Mar 16 03:21:15 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817cea2bc00 Mar 16 03:21:15 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817cea2bc00 Mar 16 03:21:15 john32 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:21:15 john29 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:21:15 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882089acd200 Mar 16 03:21:15 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882089acd200 Mar 16 03:21:15 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8829271ba800 Mar 16 03:21:15 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881640536200 Mar 16 03:21:15 john32 kernel: Lustre: dagg-OST000d-osc-ffff8817c5384000: Connection restored to 192.168.44.37@o2ib44 (at 192.168.44.37@o2ib44) Mar 16 03:21:15 john32 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:21:15 arkle3 kernel: LustreError: 294214:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881640536200 Mar 16 03:21:15 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880b5f523000 Mar 16 03:21:15 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880b5f523000 Mar 16 03:21:16 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817cea2bc00 Mar 16 03:21:16 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817cea2bc00 Mar 16 03:21:16 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880b5f523000 Mar 16 03:21:16 arkle7 kernel: LustreError: 294139:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880b5f523000 Mar 16 03:21:16 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88150d7e9400 Mar 16 03:21:16 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8815c491a200 Mar 16 03:21:16 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef8a53c00 Mar 16 03:21:16 arkle2 kernel: LustreError: 294385:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef8a53c00 Mar 16 03:21:16 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88150d7e9400 Mar 16 03:21:16 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8815c491a200 Mar 16 03:21:16 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882b83e1ac00 Mar 16 03:21:16 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882b83e1ac00 Mar 16 03:21:16 john6 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.31@o2ib44, match 1594448324140096 length 1048576 too big: 1047972 left, 1047972 allowed Mar 16 03:21:16 john6 kernel: LNetError: 913:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 8 previous similar messages Mar 16 03:21:16 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0f600 Mar 16 03:21:16 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0f600 Mar 16 03:21:16 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8815c491a000 Mar 16 03:21:16 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88150d7eac00 Mar 16 03:21:16 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882474790200 Mar 16 03:21:16 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894a8400 Mar 16 03:21:16 arkle2 kernel: LustreError: 294379:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8815c491a000 Mar 16 03:21:16 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88150d7eac00 Mar 16 03:21:16 arkle3 kernel: LustreError: 294219:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882474790200 Mar 16 03:21:16 arkle3 kernel: LustreError: 294218:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894a8400 Mar 16 03:21:16 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882ef8a53c00 Mar 16 03:21:16 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882ef8a53c00 Mar 16 03:21:16 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0f600 Mar 16 03:21:16 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0f600 Mar 16 03:21:16 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88150d7e8c00 Mar 16 03:21:16 arkle2 kernel: LustreError: 294380:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88150d7e8c00 Mar 16 03:21:16 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817cea2bc00 Mar 16 03:21:16 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817cea2bc00 Mar 16 03:21:16 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881efe3b3000 Mar 16 03:21:16 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881efe3b3000 Mar 16 03:21:16 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0f600 Mar 16 03:21:16 arkle1 kernel: LustreError: 294166:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0f600 Mar 16 03:21:16 john7 kernel: LNetError: 914:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.34@o2ib44, match 1594448319597744 length 1048576 too big: 1045464 left, 1045464 allowed Mar 16 03:21:16 john7 kernel: LNetError: 914:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 9 previous similar messages Mar 16 03:21:16 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e21a6c000 Mar 16 03:21:16 arkle4 kernel: LustreError: 294184:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e21a6c000 Mar 16 03:21:16 john31 kernel: Lustre: 936:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130832/real 1521130832] req@ffff882758f22a00 x1594699394168800/t0(0) o4->dagg-OST0003-osc-ffff882fde9fd000@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130876 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:16 john31 kernel: Lustre: 936:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 16 03:21:16 john31 kernel: Lustre: dagg-OST0003-osc-ffff882fde9fd000: Connection to dagg-OST0003 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:16 john31 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:21:16 john31 kernel: Lustre: dagg-OST0003-osc-ffff882fde9fd000: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44) Mar 16 03:21:16 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882089acd200 Mar 16 03:21:16 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882089acd200 Mar 16 03:21:16 john31 kernel: Lustre: Skipped 4 previous similar messages Mar 16 03:21:16 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881cb001d200 Mar 16 03:21:16 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881cb001d200 Mar 16 03:21:16 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881efe3b3000 Mar 16 03:21:16 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881efe3b3000 Mar 16 03:21:16 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881cb001d200 Mar 16 03:21:16 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881cb001d200 Mar 16 03:21:16 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881efe3b3000 Mar 16 03:21:16 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881efe3b3000 Mar 16 03:21:16 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882b83e1ac00 Mar 16 03:21:16 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882b83e1ac00 Mar 16 03:21:16 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e21a6aa00 Mar 16 03:21:16 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e21a6aa00 Mar 16 03:21:16 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e094c00 Mar 16 03:21:16 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e094c00 Mar 16 03:21:16 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880b5f523000 Mar 16 03:21:16 arkle7 kernel: LustreError: 294141:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880b5f523000 Mar 16 03:21:16 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88150d7e8c00 Mar 16 03:21:16 arkle2 kernel: LustreError: 294381:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88150d7e8c00 Mar 16 03:21:16 john23 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1594448282385424 length 1048576 too big: 1047668 left, 1047668 allowed Mar 16 03:21:16 john23 kernel: LNetError: 911:0:(lib-ptl.c:190:lnet_try_match_md()) Skipped 9 previous similar messages Mar 16 03:21:16 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817cea2bc00 Mar 16 03:21:16 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817cea2bc00 Mar 16 03:21:16 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881efe3b3000 Mar 16 03:21:16 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8820dec20400 Mar 16 03:21:16 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881efe3b3000 Mar 16 03:21:16 arkle1 kernel: LustreError: 294171:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8820dec20400 Mar 16 03:21:17 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880b5f523000 Mar 16 03:21:17 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880b5f523000 Mar 16 03:21:17 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881efe3b3000 Mar 16 03:21:17 arkle2 kernel: LustreError: 294382:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881efe3b3000 Mar 16 03:21:17 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882b83e1ac00 Mar 16 03:21:17 arkle5 kernel: LustreError: 294092:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882b83e1ac00 Mar 16 03:21:17 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8820035b5800 Mar 16 03:21:17 arkle7 kernel: LustreError: 294142:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8820035b5800 Mar 16 03:21:17 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894ab400 Mar 16 03:21:17 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894ab400 Mar 16 03:21:17 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882089acd200 Mar 16 03:21:17 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882089acd200 Mar 16 03:21:17 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e094c00 Mar 16 03:21:17 arkle4 kernel: LustreError: 294180:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e094c00 Mar 16 03:21:17 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882b83e1ac00 Mar 16 03:21:17 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882b83e1ac00 Mar 16 03:21:17 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808ac2ad000 Mar 16 03:21:17 arkle8 kernel: LustreError: 298957:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808ac2ad000 Mar 16 03:21:17 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882b83e1ac00 Mar 16 03:21:17 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882b83e1ac00 Mar 16 03:21:18 john10 kernel: Lustre: 943:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1521130834/real 1521130834] req@ffff882063fff800 x1594448319882256/t0(0) o4->dagg-OST0001-osc-ffff882fd4b83800@192.168.44.31@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1521130878 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:21:18 john10 kernel: Lustre: 943:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 16 03:21:18 john10 kernel: Lustre: dagg-OST0001-osc-ffff882fd4b83800: Connection to dagg-OST0001 (at 192.168.44.31@o2ib44) was lost; in progress operations using this service will wait for recovery to complete Mar 16 03:21:18 john10 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:21:18 john10 kernel: Lustre: dagg-OST0001-osc-ffff882fd4b83800: Connection restored to 192.168.44.31@o2ib44 (at 192.168.44.31@o2ib44) Mar 16 03:21:18 john10 kernel: Lustre: Skipped 3 previous similar messages Mar 16 03:21:18 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8808ac2ad000 Mar 16 03:21:18 arkle8 kernel: LustreError: 298956:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8808ac2ad000 Mar 16 03:21:18 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8820dec20e00 Mar 16 03:21:18 arkle1 kernel: LustreError: 294170:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8820dec20e00 Mar 16 03:21:18 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881640536200 Mar 16 03:21:18 arkle3 kernel: LustreError: 294216:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881640536200 Mar 16 03:21:18 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0f600 Mar 16 03:21:18 arkle1 kernel: LustreError: 294167:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0f600 Mar 16 03:21:18 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88162e097800 Mar 16 03:21:18 arkle4 kernel: LustreError: 294178:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88162e097800 Mar 16 03:21:18 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88005a839000 Mar 16 03:21:18 arkle6 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88005a839000 Mar 16 03:21:18 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882089acd200 Mar 16 03:21:18 arkle6 kernel: LustreError: 294102:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882089acd200 Mar 16 03:21:18 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0e000 Mar 16 03:21:18 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0e000 Mar 16 03:21:18 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882e22508a00 Mar 16 03:21:18 arkle4 kernel: LustreError: 294181:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882e22508a00 Mar 16 03:21:18 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8811eee46c00 Mar 16 03:21:18 arkle8 kernel: LustreError: 298954:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8811eee46c00 Mar 16 03:21:19 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8811eee46c00 Mar 16 03:21:19 arkle8 kernel: LustreError: 298952:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8811eee46c00 Mar 16 03:21:19 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817cea2bc00 Mar 16 03:21:19 arkle5 kernel: LustreError: 294088:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817cea2bc00 Mar 16 03:21:19 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882089acd200 Mar 16 03:21:19 arkle6 kernel: LustreError: 294100:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882089acd200 Mar 16 03:21:20 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882aacf5a000 Mar 16 03:21:20 arkle5 kernel: LustreError: 294093:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882aacf5a000 Mar 16 03:21:20 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8820dec26c00 Mar 16 03:21:20 arkle1 kernel: LustreError: 294168:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8820dec26c00 Mar 16 03:21:20 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880b5f526c00 Mar 16 03:21:20 arkle7 kernel: LustreError: 294140:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880b5f526c00 Mar 16 03:21:20 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881768838200 Mar 16 03:21:20 arkle2 kernel: LustreError: 294383:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881768838200 Mar 16 03:21:21 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff88268e0d3200 Mar 16 03:21:21 arkle5 kernel: LustreError: 294095:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff88268e0d3200 Mar 16 03:21:21 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8825ef547e00 Mar 16 03:21:21 arkle7 kernel: LustreError: 294143:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8825ef547e00 Mar 16 03:21:22 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882089acf800 Mar 16 03:21:22 arkle6 kernel: LustreError: 294101:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882089acf800 Mar 16 03:21:22 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882089aca000 Mar 16 03:21:22 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882089aca000 Mar 16 03:21:22 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882089aca000 Mar 16 03:21:22 arkle6 kernel: LustreError: 294099:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882089aca000 Mar 16 03:21:23 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894ab800 Mar 16 03:21:23 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894ab800 Mar 16 03:21:23 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881768838c00 Mar 16 03:21:23 arkle2 kernel: LustreError: 294384:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881768838c00 Mar 16 03:21:23 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8825ef547c00 Mar 16 03:21:23 arkle7 kernel: LustreError: 294145:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8825ef547c00 Mar 16 03:21:24 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880d91f0e000 Mar 16 03:21:24 arkle1 kernel: LustreError: 294165:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880d91f0e000 Mar 16 03:21:24 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff882c894afa00 Mar 16 03:21:24 arkle3 kernel: LustreError: 294217:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff882c894afa00 Mar 16 03:21:26 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8817cea2bc00 Mar 16 03:21:26 arkle5 kernel: LustreError: 294090:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8817cea2bc00 Mar 16 03:21:27 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff880b5f521c00 Mar 16 03:21:27 arkle7 kernel: LustreError: 294138:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff880b5f521c00