[Sun Jul 1 16:47:57 2018] Node 0 Normal: 22399*4kB (UEM) 2391*8kB (UEM) 1009*16kB (UEM) 247*32kB (UEM) 53*64kB (UEM) 32*128kB (UM) 20*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 145380kB [Sun Jul 1 16:47:57 2018] Node 1 Normal: 219940*4kB (UEM) 218484*8kB (UEM) 257678*16kB (UEM) 2507*32kB (UEM) 110*64kB (UEM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6837872kB [Sun Jul 1 16:47:57 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 16:47:57 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 16:47:57 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 16:47:57 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 16:47:57 2018] 18303614 total pagecache pages [Sun Jul 1 16:47:57 2018] 564 pages in swap cache [Sun Jul 1 16:47:57 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 16:47:57 2018] Free swap = 4180704kB [Sun Jul 1 16:47:57 2018] Total swap = 4194300kB [Sun Jul 1 16:47:57 2018] 33530455 pages RAM [Sun Jul 1 16:47:57 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 16:47:57 2018] 594386 pages reserved [Sun Jul 1 16:47:57 2018] kworker/11:3: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 16:47:57 2018] CPU: 11 PID: 122951 Comm: kworker/11:3 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 16:47:57 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 16:47:57 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 16:47:57 2018] Call Trace: [Sun Jul 1 16:47:57 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 16:47:57 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 16:47:57 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 16:47:57 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 16:47:57 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 16:47:57 2018] [] alloc_pages_current+0x98/0x110 [Sun Jul 1 16:47:57 2018] [] __get_free_pages+0xe/0x40 [Sun Jul 1 16:47:57 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Sun Jul 1 16:47:57 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Sun Jul 1 16:47:57 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 16:47:57 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 16:47:57 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 16:47:57 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 16:47:57 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 16:47:57 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 16:47:57 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 16:47:57 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 16:47:57 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 16:47:57 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 16:47:57 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 16:47:57 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 16:47:57 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 16:47:57 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 16:47:57 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 16:47:57 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 16:47:57 2018] Mem-Info: [Sun Jul 1 16:47:57 2018] active_anon:583774 inactive_anon:225556 isolated_anon:0 active_file:12136328 inactive_file:5937966 isolated_file:0 unevictable:17363 dirty:22 writeback:0 unstable:0 slab_reclaimable:7414045 slab_unreclaimable:3398958 mapped:73542 shmem:226597 pagetables:4055 bounce:0 free:1812434 free_pcp:30 free_cma:0 [Sun Jul 1 16:47:57 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 16:47:57 2018] Node 0 DMA32 free:261104kB min:1184kB low:1480kB high:1776kB active_anon:1268kB inactive_anon:8952kB active_file:62632kB inactive_file:62648kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:685744kB slab_unreclaimable:599348kB kernel_stack:192kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 16:47:57 2018] Node 0 Normal free:141512kB min:43740kB low:54672kB high:65608kB active_anon:344528kB inactive_anon:381860kB active_file:18786792kB inactive_file:18787428kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:72kB writeback:0kB mapped:151676kB shmem:551068kB slab_reclaimable:18440284kB slab_unreclaimable:5704224kB kernel_stack:3776kB pagetables:4832kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 16:47:57 2018] Node 1 Normal free:6834360kB min:45172kB low:56464kB high:67756kB active_anon:1989300kB inactive_anon:511412kB active_file:29695888kB inactive_file:4901788kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:16kB writeback:0kB mapped:141232kB shmem:352976kB slab_reclaimable:10530152kB slab_unreclaimable:7292196kB kernel_stack:10656kB pagetables:11152kB unstable:0kB bounce:0kB free_pcp:116kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 16:47:57 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 16:47:57 2018] Node 0 DMA32: 2459*4kB (UEM) 2799*8kB (UEM) 1234*16kB (UEM) 3899*32kB (UEM) 1131*64kB (UM) 94*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 261156kB [Sun Jul 1 16:47:57 2018] Node 0 Normal: 22399*4kB (UEM) 2391*8kB (UEM) 1009*16kB (UEM) 247*32kB (UEM) 53*64kB (UEM) 32*128kB (UM) 20*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 145380kB [Sun Jul 1 16:47:57 2018] Node 1 Normal: 219940*4kB (UE) 218478*8kB (UEM) 257679*16kB (UEM) 2507*32kB (UEM) 110*64kB (UEM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6837840kB [Sun Jul 1 16:47:57 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 16:47:57 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 16:47:57 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 16:47:57 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 16:47:57 2018] 18303614 total pagecache pages [Sun Jul 1 16:47:57 2018] 564 pages in swap cache [Sun Jul 1 16:47:57 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 16:47:57 2018] Free swap = 4180704kB [Sun Jul 1 16:47:57 2018] Total swap = 4194300kB [Sun Jul 1 16:47:57 2018] 33530455 pages RAM [Sun Jul 1 16:47:57 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 16:47:57 2018] 594386 pages reserved [Sun Jul 1 16:47:57 2018] kworker/11:3: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 16:47:57 2018] CPU: 11 PID: 122951 Comm: kworker/11:3 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 16:47:57 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 16:47:57 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 16:47:57 2018] Call Trace: [Sun Jul 1 16:47:57 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 16:47:57 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 16:47:57 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 16:47:57 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 16:47:57 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 16:47:57 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Sun Jul 1 16:47:57 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Sun Jul 1 16:47:57 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 16:47:57 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 16:47:57 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 16:47:57 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 16:47:57 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 16:47:57 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 16:47:57 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 16:47:57 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 16:47:57 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 16:47:57 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 16:47:57 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 16:47:57 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 16:47:57 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 16:47:57 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 16:47:57 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 16:47:57 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 16:47:57 2018] Mem-Info: [Sun Jul 1 16:47:57 2018] active_anon:583774 inactive_anon:225556 isolated_anon:0 active_file:12136328 inactive_file:5937966 isolated_file:0 unevictable:17363 dirty:22 writeback:0 unstable:0 slab_reclaimable:7413805 slab_unreclaimable:3398958 mapped:73542 shmem:226597 pagetables:4055 bounce:0 free:1812679 free_pcp:29 free_cma:0 [Sun Jul 1 16:47:57 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 16:47:57 2018] Node 0 DMA32 free:261104kB min:1184kB low:1480kB high:1776kB active_anon:1268kB inactive_anon:8952kB active_file:62632kB inactive_file:62648kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:685744kB slab_unreclaimable:599348kB kernel_stack:192kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 16:47:57 2018] Node 0 Normal free:141512kB min:43740kB low:54672kB high:65608kB active_anon:344528kB inactive_anon:381860kB active_file:18786792kB inactive_file:18787428kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:72kB writeback:0kB mapped:151676kB shmem:551068kB slab_reclaimable:18440284kB slab_unreclaimable:5704224kB kernel_stack:3776kB pagetables:4832kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 16:47:57 2018] Node 1 Normal free:6835340kB min:45172kB low:56464kB high:67756kB active_anon:1989300kB inactive_anon:511412kB active_file:29695888kB inactive_file:4901788kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:16kB writeback:0kB mapped:141232kB shmem:352976kB slab_reclaimable:10529192kB slab_unreclaimable:7292196kB kernel_stack:10656kB pagetables:11152kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 16:47:57 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 16:47:57 2018] Node 0 DMA32: 2459*4kB (UEM) 2799*8kB (UEM) 1234*16kB (UEM) 3899*32kB (UEM) 1131*64kB (UM) 94*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 261156kB [Sun Jul 1 16:47:57 2018] Node 0 Normal: 22401*4kB (UEM) 2392*8kB (UEM) 1010*16kB (UEM) 247*32kB (UEM) 53*64kB (UEM) 32*128kB (UM) 20*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 145412kB [Sun Jul 1 16:47:57 2018] Node 1 Normal: 219944*4kB (UEM) 218470*8kB (UEM) 257682*16kB (UEM) 2528*32kB (UEM) 116*64kB (UEM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6838896kB [Sun Jul 1 16:47:57 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 16:47:57 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 16:47:57 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 16:47:57 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 16:47:57 2018] 18303614 total pagecache pages [Sun Jul 1 16:47:57 2018] 564 pages in swap cache [Sun Jul 1 16:47:57 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 16:47:57 2018] Free swap = 4180704kB [Sun Jul 1 16:47:57 2018] Total swap = 4194300kB [Sun Jul 1 16:47:57 2018] 33530455 pages RAM [Sun Jul 1 16:47:57 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 16:47:57 2018] 594386 pages reserved [Sun Jul 1 16:47:57 2018] kworker/11:3: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 16:47:57 2018] CPU: 11 PID: 122951 Comm: kworker/11:3 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 16:47:57 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 16:47:57 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 16:47:57 2018] Call Trace: [Sun Jul 1 16:47:57 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 16:47:57 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 16:47:57 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 16:47:57 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 16:47:57 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 16:47:57 2018] [] alloc_pages_current+0x98/0x110 [Sun Jul 1 16:47:57 2018] [] __get_free_pages+0xe/0x40 [Sun Jul 1 16:47:57 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Sun Jul 1 16:47:57 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Sun Jul 1 16:47:57 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 16:47:57 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 16:47:57 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 16:47:57 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 16:47:57 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 16:47:57 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 16:47:57 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 16:47:57 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 16:47:57 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 16:47:57 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 16:47:57 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 16:47:57 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 16:47:57 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 16:47:57 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 16:47:57 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 16:47:57 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 16:47:57 2018] Mem-Info: [Sun Jul 1 16:47:57 2018] active_anon:583892 inactive_anon:225556 isolated_anon:0 active_file:12136328 inactive_file:5937966 isolated_file:0 unevictable:17363 dirty:22 writeback:0 unstable:0 slab_reclaimable:7413805 slab_unreclaimable:3398958 mapped:73542 shmem:226597 pagetables:4055 bounce:0 free:1812679 free_pcp:36 free_cma:0 [Sun Jul 1 16:47:57 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 16:47:57 2018] Node 0 DMA32 free:261104kB min:1184kB low:1480kB high:1776kB active_anon:1268kB inactive_anon:8952kB active_file:62632kB inactive_file:62648kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:685744kB slab_unreclaimable:599348kB kernel_stack:192kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 16:47:57 2018] Node 0 Normal free:141512kB min:43740kB low:54672kB high:65608kB active_anon:344528kB inactive_anon:381860kB active_file:18786792kB inactive_file:18787428kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:72kB writeback:0kB mapped:151676kB shmem:551068kB slab_reclaimable:18440284kB slab_unreclaimable:5704224kB kernel_stack:3776kB pagetables:4832kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 16:47:57 2018] Node 1 Normal free:6835340kB min:45172kB low:56464kB high:67756kB active_anon:1989772kB inactive_anon:511412kB active_file:29695888kB inactive_file:4901788kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:16kB writeback:0kB mapped:141232kB shmem:352976kB slab_reclaimable:10529192kB slab_unreclaimable:7292196kB kernel_stack:10656kB pagetables:11152kB unstable:0kB bounce:0kB free_pcp:240kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 16:47:57 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 16:47:57 2018] Node 0 DMA32: 2459*4kB (UEM) 2799*8kB (UEM) 1234*16kB (UEM) 3899*32kB (UEM) 1131*64kB (UM) 94*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 261156kB [Sun Jul 1 16:47:57 2018] Node 0 Normal: 22401*4kB (UEM) 2392*8kB (UEM) 1010*16kB (UEM) 247*32kB (UEM) 53*64kB (UEM) 32*128kB (UM) 20*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 145412kB [Sun Jul 1 16:47:57 2018] Node 1 Normal: 219922*4kB (UE) 218462*8kB (UEM) 257677*16kB (UEM) 2514*32kB (UEM) 116*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6838088kB [Sun Jul 1 16:47:57 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 16:47:57 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 16:47:57 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 16:47:57 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 16:47:57 2018] 18303614 total pagecache pages [Sun Jul 1 16:47:57 2018] 564 pages in swap cache [Sun Jul 1 16:47:57 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 16:47:57 2018] Free swap = 4180704kB [Sun Jul 1 16:47:57 2018] Total swap = 4194300kB [Sun Jul 1 16:47:57 2018] 33530455 pages RAM [Sun Jul 1 16:47:57 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 16:47:57 2018] 594386 pages reserved [Sun Jul 1 16:47:57 2018] kworker/11:3: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 16:47:57 2018] CPU: 11 PID: 122951 Comm: kworker/11:3 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 16:47:57 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 16:47:57 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 16:47:57 2018] Call Trace: [Sun Jul 1 16:47:57 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 16:47:57 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 16:47:57 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 16:47:57 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 16:47:57 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Sun Jul 1 16:47:57 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Sun Jul 1 16:47:57 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 16:47:57 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 16:47:57 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 16:47:57 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 16:47:57 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 16:47:57 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 16:47:57 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 16:47:57 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 16:47:57 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 16:47:57 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 16:47:57 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 16:47:57 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 16:47:57 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 16:47:57 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 16:47:57 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 16:47:57 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 16:47:57 2018] Mem-Info: [Sun Jul 1 16:47:57 2018] active_anon:579884 inactive_anon:225556 isolated_anon:0 active_file:12136335 inactive_file:5937969 isolated_file:2 unevictable:17363 dirty:22 writeback:0 unstable:0 slab_reclaimable:7413805 slab_unreclaimable:3398827 mapped:73545 shmem:226597 pagetables:4061 bounce:0 free:1816988 free_pcp:435 free_cma:0 [Sun Jul 1 16:47:57 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 16:47:57 2018] Node 0 DMA32 free:261100kB min:1184kB low:1480kB high:1776kB active_anon:1268kB inactive_anon:8952kB active_file:62648kB inactive_file:62648kB unevictable:3464kB isolated(anon):0kB isolated(file):8kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:685744kB slab_unreclaimable:599348kB kernel_stack:192kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 16:47:57 2018] Node 0 Normal free:142172kB min:43740kB low:54672kB high:65608kB active_anon:344544kB inactive_anon:381860kB active_file:18786804kB inactive_file:18787440kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:72kB writeback:0kB mapped:151676kB shmem:551068kB slab_reclaimable:18440284kB slab_unreclaimable:5703768kB kernel_stack:3776kB pagetables:4856kB unstable:0kB bounce:0kB free_pcp:1068kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 16:47:57 2018] Node 1 Normal free:6851920kB min:45172kB low:56464kB high:67756kB active_anon:1973724kB inactive_anon:511412kB active_file:29695888kB inactive_file:4901788kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:16kB writeback:0kB mapped:141244kB shmem:352976kB slab_reclaimable:10529192kB slab_unreclaimable:7292128kB kernel_stack:10656kB pagetables:11152kB unstable:0kB bounce:0kB free_pcp:672kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 16:47:57 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 16:47:57 2018] Node 0 DMA32: 2077*4kB (UEM) 2731*8kB (UEM) 1246*16kB (UEM) 3926*32kB (UEM) 1139*64kB (UM) 98*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 261164kB [Sun Jul 1 16:47:57 2018] Node 0 Normal: 20167*4kB (UEM) 1761*8kB (UEM) 833*16kB (UEM) 396*32kB (UEM) 136*64kB (UEM) 66*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 144820kB [Sun Jul 1 16:47:57 2018] Node 1 Normal: 219885*4kB (UEM) 218727*8kB (UEM) 258007*16kB (UEM) 2678*32kB (UEM) 161*64kB (UEM) 4*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6853980kB [Sun Jul 1 16:47:57 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 16:47:57 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 16:47:57 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 16:47:57 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 16:47:57 2018] 18303615 total pagecache pages [Sun Jul 1 16:47:57 2018] 564 pages in swap cache [Sun Jul 1 16:47:57 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 16:47:57 2018] Free swap = 4180704kB [Sun Jul 1 16:47:57 2018] Total swap = 4194300kB [Sun Jul 1 16:47:57 2018] 33530455 pages RAM [Sun Jul 1 16:47:57 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 16:47:57 2018] 594386 pages reserved [Sun Jul 1 16:47:57 2018] kworker/11:3: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 16:47:57 2018] CPU: 11 PID: 122951 Comm: kworker/11:3 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 16:47:57 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 16:47:57 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 16:47:57 2018] Call Trace: [Sun Jul 1 16:47:57 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 16:47:57 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 16:47:57 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 16:47:57 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 16:47:57 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 16:47:57 2018] [] alloc_pages_current+0x98/0x110 [Sun Jul 1 16:47:57 2018] [] __get_free_pages+0xe/0x40 [Sun Jul 1 16:47:57 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Sun Jul 1 16:47:57 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Sun Jul 1 16:47:57 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 16:47:57 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 16:47:57 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 16:47:57 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 16:47:57 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 16:47:57 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 16:47:57 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 16:47:57 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 16:47:57 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 16:47:57 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 16:47:57 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 16:47:57 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 16:47:57 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 16:47:57 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 16:47:57 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 16:47:57 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 16:47:57 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 16:47:57 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 16:47:57 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 16:47:57 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 16:47:57 2018] Mem-Info: [Sun Jul 1 16:47:57 2018] active_anon:579884 inactive_anon:225556 isolated_anon:0 active_file:12136335 inactive_file:5937969 isolated_file:2 unevictable:17363 dirty:22 writeback:0 unstable:0 slab_reclaimable:7413805 slab_unreclaimable:3398827 mapped:73545 shmem:226597 pagetables:4061 bounce:0 free:1816988 free_pcp:34 free_cma:0 [Sun Jul 1 16:47:57 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 16:47:57 2018] Node 0 DMA32 free:261100kB min:1184kB low:1480kB high:1776kB active_anon:1268kB inactive_anon:8952kB active_file:62648kB inactive_file:62648kB unevictable:3464kB isolated(anon):0kB isolated(file):8kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:685744kB slab_unreclaimable:599348kB kernel_stack:192kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 16:47:57 2018] Node 0 Normal free:142172kB min:43740kB low:54672kB high:65608kB active_anon:344544kB inactive_anon:381860kB active_file:18786804kB inactive_file:18787440kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:72kB writeback:0kB mapped:151676kB shmem:551068kB slab_reclaimable:18440284kB slab_unreclaimable:5703768kB kernel_stack:3776kB pagetables:4856kB unstable:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 16:47:57 2018] Node 1 Normal free:6851920kB min:45172kB low:56464kB high:67756kB active_anon:1973724kB inactive_anon:511412kB active_file:29695888kB inactive_file:4901788kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:16kB writeback:0kB mapped:141244kB shmem:352976kB slab_reclaimable:10529192kB slab_unreclaimable:7292128kB kernel_stack:10656kB pagetables:11152kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 16:47:57 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 16:47:57 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 16:47:57 2018] Node 0 DMA32: 2077*4kB (UEM) 2731*8kB (UEM) 1246*16kB (UEM) 3926*32kB (UEM) 1139*64kB (UM) 98*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 261164kB [Sun Jul 1 16:47:57 2018] Node 0 Normal: 20434*4kB (UEM) 1761*8kB (UEM) 833*16kB (UEM) 396*32kB (UEM) 136*64kB (UEM) 66*128kB (UM) 27*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 145888kB [Sun Jul 1 16:47:57 2018] Node 1 Normal: 219918*4kB (UEM) 218775*8kB (UEM) 258009*16kB (UEM) 2678*32kB (UEM) 162*64kB (UEM) 4*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6854592kB [Sun Jul 1 16:47:57 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 16:47:57 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 16:47:57 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 16:47:57 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 16:47:57 2018] 18303615 total pagecache pages [Sun Jul 1 16:47:57 2018] 564 pages in swap cache [Sun Jul 1 16:47:57 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 16:47:57 2018] Free swap = 4180704kB [Sun Jul 1 16:47:57 2018] Total swap = 4194300kB [Sun Jul 1 16:47:57 2018] 33530455 pages RAM [Sun Jul 1 16:47:57 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 16:47:57 2018] 594386 pages reserved [Sun Jul 1 16:47:57 2018] LNetError: 122951:0:(o2iblnd.c:934:kiblnd_create_conn()) Can't create QP: -12, send_wr: 409, recv_wr: 4, send_sge: 30, recv_sge: 1 [Sun Jul 1 16:47:57 2018] LNetError: 122951:0:(o2iblnd.c:934:kiblnd_create_conn()) Skipped 1 previous similar message [Sun Jul 1 16:55:51 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 16:55:51 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) Skipped 1954585 previous similar messages [Sun Jul 1 16:55:51 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 16:55:51 2018] Lustre: Skipped 1954350 previous similar messages [Sun Jul 1 16:55:51 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 16:55:51 2018] Lustre: Skipped 1954350 previous similar messages [Sun Jul 1 17:05:51 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 17:05:51 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) Skipped 2410831 previous similar messages [Sun Jul 1 17:05:51 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 17:05:51 2018] Lustre: Skipped 2410831 previous similar messages [Sun Jul 1 17:05:51 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 17:05:51 2018] Lustre: Skipped 2410831 previous similar messages [Sun Jul 1 17:15:51 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 17:15:51 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) Skipped 1935505 previous similar messages [Sun Jul 1 17:15:51 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 17:15:51 2018] Lustre: Skipped 1935505 previous similar messages [Sun Jul 1 17:15:51 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 17:15:51 2018] Lustre: Skipped 1935505 previous similar messages [Sun Jul 1 17:25:51 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 17:25:51 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 1934079 previous similar messages [Sun Jul 1 17:25:51 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 17:25:51 2018] Lustre: Skipped 1934079 previous similar messages [Sun Jul 1 17:25:51 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 17:25:51 2018] Lustre: Skipped 1934079 previous similar messages [Sun Jul 1 17:35:52 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 17:35:52 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) Skipped 2400211 previous similar messages [Sun Jul 1 17:35:52 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 17:35:52 2018] Lustre: Skipped 2400211 previous similar messages [Sun Jul 1 17:35:52 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 17:35:52 2018] Lustre: Skipped 2400211 previous similar messages [Sun Jul 1 17:45:52 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 17:45:52 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) Skipped 1986622 previous similar messages [Sun Jul 1 17:45:52 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 17:45:52 2018] Lustre: Skipped 1986622 previous similar messages [Sun Jul 1 17:45:52 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 17:45:52 2018] Lustre: Skipped 1986622 previous similar messages [Sun Jul 1 17:55:52 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 17:55:52 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) Skipped 1886630 previous similar messages [Sun Jul 1 17:55:52 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 17:55:52 2018] Lustre: Skipped 1886630 previous similar messages [Sun Jul 1 17:55:52 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 17:55:52 2018] Lustre: Skipped 1886630 previous similar messages [Sun Jul 1 18:05:52 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 18:05:52 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) Skipped 2377064 previous similar messages [Sun Jul 1 18:05:52 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 18:05:52 2018] Lustre: Skipped 2377064 previous similar messages [Sun Jul 1 18:05:52 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 18:05:52 2018] Lustre: Skipped 2377064 previous similar messages [Sun Jul 1 18:15:52 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 18:15:52 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) Skipped 1923858 previous similar messages [Sun Jul 1 18:15:52 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 18:15:52 2018] Lustre: Skipped 1923858 previous similar messages [Sun Jul 1 18:15:52 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 18:15:52 2018] Lustre: Skipped 1923858 previous similar messages [Sun Jul 1 18:26:11 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 18:26:11 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) Skipped 574558 previous similar messages [Sun Jul 1 18:26:11 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 18:26:11 2018] Lustre: Skipped 574558 previous similar messages [Sun Jul 1 18:26:11 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 18:26:11 2018] Lustre: Skipped 574558 previous similar messages [Sun Jul 1 18:36:13 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 18:36:13 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) Skipped 982179 previous similar messages [Sun Jul 1 18:36:13 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 18:36:13 2018] Lustre: Skipped 982179 previous similar messages [Sun Jul 1 18:36:13 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 18:36:13 2018] Lustre: Skipped 982179 previous similar messages [Sun Jul 1 18:46:13 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 18:46:13 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) Skipped 977211 previous similar messages [Sun Jul 1 18:46:13 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 18:46:13 2018] Lustre: Skipped 977211 previous similar messages [Sun Jul 1 18:46:13 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 18:46:13 2018] Lustre: Skipped 977211 previous similar messages [Sun Jul 1 18:56:13 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 18:56:13 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) Skipped 971512 previous similar messages [Sun Jul 1 18:56:13 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 18:56:13 2018] Lustre: Skipped 971804 previous similar messages [Sun Jul 1 18:56:13 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 18:56:13 2018] Lustre: Skipped 971804 previous similar messages [Sun Jul 1 19:06:13 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 19:06:13 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) Skipped 965174 previous similar messages [Sun Jul 1 19:06:13 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 19:06:13 2018] Lustre: Skipped 965211 previous similar messages [Sun Jul 1 19:06:13 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 19:06:13 2018] Lustre: Skipped 965211 previous similar messages [Sun Jul 1 19:17:16 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 19:17:16 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) Skipped 1284586 previous similar messages [Sun Jul 1 19:17:16 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 19:17:16 2018] Lustre: Skipped 1284257 previous similar messages [Sun Jul 1 19:17:16 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 19:17:16 2018] Lustre: Skipped 1284257 previous similar messages [Sun Jul 1 19:27:16 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 19:27:16 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) Skipped 1445200 previous similar messages [Sun Jul 1 19:27:16 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 19:27:16 2018] Lustre: Skipped 1445240 previous similar messages [Sun Jul 1 19:27:16 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 19:27:16 2018] Lustre: Skipped 1445240 previous similar messages [Sun Jul 1 19:37:16 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 19:37:16 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) Skipped 2360728 previous similar messages [Sun Jul 1 19:37:17 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 19:37:17 2018] Lustre: Skipped 2360688 previous similar messages [Sun Jul 1 19:37:17 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 19:37:17 2018] Lustre: Skipped 2360688 previous similar messages [Sun Jul 1 19:47:16 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 19:47:17 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 2433473 previous similar messages [Sun Jul 1 19:47:17 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 19:47:17 2018] Lustre: Skipped 2433473 previous similar messages [Sun Jul 1 19:47:17 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 19:47:17 2018] Lustre: Skipped 2433473 previous similar messages [Sun Jul 1 19:57:17 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 19:57:17 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) Skipped 1918097 previous similar messages [Sun Jul 1 19:57:17 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 19:57:17 2018] Lustre: Skipped 1918097 previous similar messages [Sun Jul 1 19:57:17 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 19:57:17 2018] Lustre: Skipped 1918097 previous similar messages [Sun Jul 1 20:07:17 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 20:07:17 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) Skipped 2407839 previous similar messages [Sun Jul 1 20:07:17 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 20:07:17 2018] Lustre: Skipped 2408053 previous similar messages [Sun Jul 1 20:07:17 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 20:07:17 2018] Lustre: Skipped 2408053 previous similar messages [Sun Jul 1 20:17:17 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 20:17:17 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) Skipped 2403788 previous similar messages [Sun Jul 1 20:17:17 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 20:17:17 2018] Lustre: Skipped 2403786 previous similar messages [Sun Jul 1 20:17:17 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 20:17:17 2018] Lustre: Skipped 2403786 previous similar messages [Sun Jul 1 20:27:17 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 20:27:17 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) Skipped 965660 previous similar messages [Sun Jul 1 20:27:17 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 20:27:17 2018] Lustre: Skipped 965754 previous similar messages [Sun Jul 1 20:27:17 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 20:27:17 2018] Lustre: Skipped 965754 previous similar messages [Sun Jul 1 20:37:17 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 20:37:17 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) Skipped 2429000 previous similar messages [Sun Jul 1 20:37:17 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 20:37:17 2018] Lustre: Skipped 2428981 previous similar messages [Sun Jul 1 20:37:17 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 20:37:17 2018] Lustre: Skipped 2428981 previous similar messages [Sun Jul 1 20:47:17 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 20:47:17 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) Skipped 2427966 previous similar messages [Sun Jul 1 20:47:17 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 20:47:17 2018] Lustre: Skipped 2427966 previous similar messages [Sun Jul 1 20:47:17 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 20:47:17 2018] Lustre: Skipped 2427966 previous similar messages [Sun Jul 1 20:57:17 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 20:57:17 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) Skipped 490884 previous similar messages [Sun Jul 1 20:57:17 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 20:57:17 2018] Lustre: Skipped 490878 previous similar messages [Sun Jul 1 20:57:17 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 20:57:17 2018] Lustre: Skipped 490878 previous similar messages [Sun Jul 1 21:07:33 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 21:07:33 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 1526344 previous similar messages [Sun Jul 1 21:07:33 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 21:07:33 2018] Lustre: Skipped 1526063 previous similar messages [Sun Jul 1 21:07:33 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 21:07:33 2018] Lustre: Skipped 1526063 previous similar messages [Sun Jul 1 21:17:44 2018] warn_alloc_failed: 22 callbacks suppressed [Sun Jul 1 21:17:44 2018] kworker/11:2: page allocation failure: order:8, mode:0x80d0 [Sun Jul 1 21:17:44 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 21:17:44 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 21:17:44 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 21:17:44 2018] Call Trace: [Sun Jul 1 21:17:44 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 21:17:44 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 21:17:44 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Sun Jul 1 21:17:44 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Sun Jul 1 21:17:44 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 21:17:44 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 21:17:44 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 21:17:44 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 21:17:44 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 21:17:44 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 21:17:44 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 21:17:44 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 21:17:44 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 21:17:44 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 21:17:44 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] Mem-Info: [Sun Jul 1 21:17:44 2018] active_anon:584597 inactive_anon:225747 isolated_anon:0 active_file:12229779 inactive_file:5974029 isolated_file:2 unevictable:17363 dirty:280 writeback:0 unstable:0 slab_reclaimable:7216416 slab_unreclaimable:3439641 mapped:73724 shmem:226598 pagetables:4192 bounce:0 free:1838419 free_pcp:0 free_cma:0 [Sun Jul 1 21:17:44 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 21:17:44 2018] Node 0 DMA32 free:300368kB min:1184kB low:1480kB high:1776kB active_anon:1252kB inactive_anon:8952kB active_file:41720kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):8kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:44kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694840kB slab_unreclaimable:594964kB kernel_stack:192kB pagetables:268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 21:17:44 2018] Node 0 Normal free:200640kB min:43740kB low:54672kB high:65608kB active_anon:343836kB inactive_anon:381644kB active_file:18869916kB inactive_file:18749108kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:528kB writeback:0kB mapped:149312kB shmem:551068kB slab_reclaimable:18460252kB slab_unreclaimable:5580340kB kernel_stack:4016kB pagetables:5104kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 1 Normal free:6839908kB min:45172kB low:56464kB high:67756kB active_anon:1993300kB inactive_anon:512392kB active_file:30007480kB inactive_file:5107396kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:548kB writeback:0kB mapped:144324kB shmem:352980kB slab_reclaimable:9710572kB slab_unreclaimable:7583196kB kernel_stack:10432kB pagetables:11396kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 21:17:44 2018] Node 0 DMA32: 2535*4kB (UEM) 3737*8kB (UEM) 1929*16kB (UEM) 3678*32kB (UEM) 1402*64kB (UEM) 167*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 300468kB [Sun Jul 1 21:17:44 2018] Node 0 Normal: 26864*4kB (UEM) 6077*8kB (UEM) 1281*16kB (UEM) 616*32kB (UEM) 90*64kB (UEM) 13*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 203960kB [Sun Jul 1 21:17:44 2018] Node 1 Normal: 227630*4kB (UEM) 234308*8kB (UEM) 239175*16kB (UEM) 7104*32kB (UEM) 73*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6843784kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] 18433225 total pagecache pages [Sun Jul 1 21:17:44 2018] 564 pages in swap cache [Sun Jul 1 21:17:44 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 21:17:44 2018] Free swap = 4180704kB [Sun Jul 1 21:17:44 2018] Total swap = 4194300kB [Sun Jul 1 21:17:44 2018] 33530455 pages RAM [Sun Jul 1 21:17:44 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 21:17:44 2018] 594386 pages reserved [Sun Jul 1 21:17:44 2018] kworker/11:2: page allocation failure: order:8, mode:0x80d0 [Sun Jul 1 21:17:44 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 21:17:44 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 21:17:44 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 21:17:44 2018] Call Trace: [Sun Jul 1 21:17:44 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 21:17:44 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 21:17:44 2018] [] alloc_pages_current+0x98/0x110 [Sun Jul 1 21:17:44 2018] [] __get_free_pages+0xe/0x40 [Sun Jul 1 21:17:44 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Sun Jul 1 21:17:44 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Sun Jul 1 21:17:44 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 21:17:44 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 21:17:44 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 21:17:44 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 21:17:44 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 21:17:44 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 21:17:44 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 21:17:44 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 21:17:44 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 21:17:44 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 21:17:44 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] Mem-Info: [Sun Jul 1 21:17:44 2018] active_anon:584597 inactive_anon:225747 isolated_anon:0 active_file:12229779 inactive_file:5974029 isolated_file:2 unevictable:17363 dirty:280 writeback:0 unstable:0 slab_reclaimable:7216416 slab_unreclaimable:3439641 mapped:73724 shmem:226598 pagetables:4192 bounce:0 free:1838273 free_pcp:185 free_cma:0 [Sun Jul 1 21:17:44 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 21:17:44 2018] Node 0 DMA32 free:300368kB min:1184kB low:1480kB high:1776kB active_anon:1252kB inactive_anon:8952kB active_file:41720kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):8kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:44kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694840kB slab_unreclaimable:594964kB kernel_stack:192kB pagetables:268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 21:17:44 2018] Node 0 Normal free:200640kB min:43740kB low:54672kB high:65608kB active_anon:343836kB inactive_anon:381644kB active_file:18869916kB inactive_file:18749108kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:528kB writeback:0kB mapped:149312kB shmem:551068kB slab_reclaimable:18460252kB slab_unreclaimable:5580340kB kernel_stack:4016kB pagetables:5104kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 1 Normal free:6839324kB min:45172kB low:56464kB high:67756kB active_anon:1993300kB inactive_anon:512392kB active_file:30007480kB inactive_file:5107396kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:548kB writeback:0kB mapped:144324kB shmem:352980kB slab_reclaimable:9710572kB slab_unreclaimable:7583196kB kernel_stack:10432kB pagetables:11396kB unstable:0kB bounce:0kB free_pcp:740kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 21:17:44 2018] Node 0 DMA32: 2541*4kB (UEM) 3736*8kB (UEM) 1928*16kB (UEM) 3678*32kB (UEM) 1402*64kB (UEM) 167*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 300468kB [Sun Jul 1 21:17:44 2018] Node 0 Normal: 26866*4kB (UEM) 6078*8kB (UEM) 1281*16kB (UEM) 616*32kB (UEM) 90*64kB (UEM) 13*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 203976kB [Sun Jul 1 21:17:44 2018] Node 1 Normal: 227444*4kB (UEM) 234308*8kB (UEM) 239176*16kB (UEM) 7104*32kB (UEM) 73*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6843056kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] 18433225 total pagecache pages [Sun Jul 1 21:17:44 2018] 564 pages in swap cache [Sun Jul 1 21:17:44 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 21:17:44 2018] Free swap = 4180704kB [Sun Jul 1 21:17:44 2018] Total swap = 4194300kB [Sun Jul 1 21:17:44 2018] 33530455 pages RAM [Sun Jul 1 21:17:44 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 21:17:44 2018] 594386 pages reserved [Sun Jul 1 21:17:44 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 21:17:44 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 21:17:44 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 21:17:44 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 21:17:44 2018] Call Trace: [Sun Jul 1 21:17:44 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 21:17:44 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 21:17:44 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 21:17:44 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Sun Jul 1 21:17:44 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Sun Jul 1 21:17:44 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 21:17:44 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 21:17:44 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 21:17:44 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 21:17:44 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 21:17:44 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 21:17:44 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 21:17:44 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 21:17:44 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 21:17:44 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 21:17:44 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] Mem-Info: [Sun Jul 1 21:17:44 2018] active_anon:584597 inactive_anon:225747 isolated_anon:0 active_file:12229779 inactive_file:5974029 isolated_file:2 unevictable:17363 dirty:280 writeback:0 unstable:0 slab_reclaimable:7216416 slab_unreclaimable:3439641 mapped:73724 shmem:226598 pagetables:4192 bounce:0 free:1838391 free_pcp:0 free_cma:0 [Sun Jul 1 21:17:44 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 21:17:44 2018] Node 0 DMA32 free:300368kB min:1184kB low:1480kB high:1776kB active_anon:1252kB inactive_anon:8952kB active_file:41720kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):8kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:44kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694840kB slab_unreclaimable:594964kB kernel_stack:192kB pagetables:268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 21:17:44 2018] Node 0 Normal free:200640kB min:43740kB low:54672kB high:65608kB active_anon:343836kB inactive_anon:381644kB active_file:18869916kB inactive_file:18749108kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:528kB writeback:0kB mapped:149312kB shmem:551068kB slab_reclaimable:18460252kB slab_unreclaimable:5580340kB kernel_stack:4016kB pagetables:5104kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 1 Normal free:6839796kB min:45172kB low:56464kB high:67756kB active_anon:1993300kB inactive_anon:512392kB active_file:30007480kB inactive_file:5107396kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:548kB writeback:0kB mapped:144324kB shmem:352980kB slab_reclaimable:9710572kB slab_unreclaimable:7583196kB kernel_stack:10432kB pagetables:11396kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 21:17:44 2018] Node 0 DMA32: 2541*4kB (UEM) 3736*8kB (UEM) 1928*16kB (UEM) 3678*32kB (UEM) 1402*64kB (UEM) 167*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 300468kB [Sun Jul 1 21:17:44 2018] Node 0 Normal: 26866*4kB (UEM) 6078*8kB (UEM) 1281*16kB (UEM) 616*32kB (UEM) 90*64kB (UEM) 13*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 203976kB [Sun Jul 1 21:17:44 2018] Node 1 Normal: 227626*4kB (UEM) 234308*8kB (UEM) 239176*16kB (UEM) 7104*32kB (UEM) 73*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6843784kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] 18433225 total pagecache pages [Sun Jul 1 21:17:44 2018] 564 pages in swap cache [Sun Jul 1 21:17:44 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 21:17:44 2018] Free swap = 4180704kB [Sun Jul 1 21:17:44 2018] Total swap = 4194300kB [Sun Jul 1 21:17:44 2018] 33530455 pages RAM [Sun Jul 1 21:17:44 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 21:17:44 2018] 594386 pages reserved [Sun Jul 1 21:17:44 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 21:17:44 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 21:17:44 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 21:17:44 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 21:17:44 2018] Call Trace: [Sun Jul 1 21:17:44 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 21:17:44 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 21:17:44 2018] [] alloc_pages_current+0x98/0x110 [Sun Jul 1 21:17:44 2018] [] __get_free_pages+0xe/0x40 [Sun Jul 1 21:17:44 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Sun Jul 1 21:17:44 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Sun Jul 1 21:17:44 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 21:17:44 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 21:17:44 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 21:17:44 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 21:17:44 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 21:17:44 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 21:17:44 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 21:17:44 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 21:17:44 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 21:17:44 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 21:17:44 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] Mem-Info: [Sun Jul 1 21:17:44 2018] active_anon:584597 inactive_anon:225747 isolated_anon:0 active_file:12229783 inactive_file:5974029 isolated_file:2 unevictable:17363 dirty:280 writeback:0 unstable:0 slab_reclaimable:7216416 slab_unreclaimable:3439641 mapped:73724 shmem:226598 pagetables:4192 bounce:0 free:1838396 free_pcp:184 free_cma:0 [Sun Jul 1 21:17:44 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 21:17:44 2018] Node 0 DMA32 free:300400kB min:1184kB low:1480kB high:1776kB active_anon:1252kB inactive_anon:8952kB active_file:41736kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):8kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:44kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694840kB slab_unreclaimable:594964kB kernel_stack:192kB pagetables:268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 21:17:44 2018] Node 0 Normal free:200640kB min:43740kB low:54672kB high:65608kB active_anon:343836kB inactive_anon:381644kB active_file:18869916kB inactive_file:18749108kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:528kB writeback:0kB mapped:149312kB shmem:551068kB slab_reclaimable:18460252kB slab_unreclaimable:5580340kB kernel_stack:4016kB pagetables:5104kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 1 Normal free:6839784kB min:45172kB low:56464kB high:67756kB active_anon:1993300kB inactive_anon:512392kB active_file:30007480kB inactive_file:5107396kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:548kB writeback:0kB mapped:144324kB shmem:352980kB slab_reclaimable:9710572kB slab_unreclaimable:7583196kB kernel_stack:10432kB pagetables:11396kB unstable:0kB bounce:0kB free_pcp:736kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 21:17:44 2018] Node 0 DMA32: 2541*4kB (UEM) 3738*8kB (UEM) 1929*16kB (UEM) 3678*32kB (UEM) 1402*64kB (UEM) 167*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 300500kB [Sun Jul 1 21:17:44 2018] Node 0 Normal: 26871*4kB (UEM) 6078*8kB (UEM) 1281*16kB (UEM) 616*32kB (UEM) 90*64kB (UEM) 13*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 203996kB [Sun Jul 1 21:17:44 2018] Node 1 Normal: 227434*4kB (UEM) 234307*8kB (UEM) 239174*16kB (UEM) 7106*32kB (UEM) 73*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6843040kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] 18433225 total pagecache pages [Sun Jul 1 21:17:44 2018] 564 pages in swap cache [Sun Jul 1 21:17:44 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 21:17:44 2018] Free swap = 4180704kB [Sun Jul 1 21:17:44 2018] Total swap = 4194300kB [Sun Jul 1 21:17:44 2018] 33530455 pages RAM [Sun Jul 1 21:17:44 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 21:17:44 2018] 594386 pages reserved [Sun Jul 1 21:17:44 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 21:17:44 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 21:17:44 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 21:17:44 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 21:17:44 2018] Call Trace: [Sun Jul 1 21:17:44 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 21:17:44 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 21:17:44 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 21:17:44 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Sun Jul 1 21:17:44 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Sun Jul 1 21:17:44 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 21:17:44 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 21:17:44 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 21:17:44 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 21:17:44 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 21:17:44 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 21:17:44 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 21:17:44 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 21:17:44 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 21:17:44 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 21:17:44 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] Mem-Info: [Sun Jul 1 21:17:44 2018] active_anon:584597 inactive_anon:225747 isolated_anon:0 active_file:12229783 inactive_file:5974029 isolated_file:2 unevictable:17363 dirty:280 writeback:0 unstable:0 slab_reclaimable:7216416 slab_unreclaimable:3439641 mapped:73724 shmem:226598 pagetables:4192 bounce:0 free:1838514 free_pcp:2 free_cma:0 [Sun Jul 1 21:17:44 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 21:17:44 2018] Node 0 DMA32 free:300400kB min:1184kB low:1480kB high:1776kB active_anon:1252kB inactive_anon:8952kB active_file:41736kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):8kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:44kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694840kB slab_unreclaimable:594964kB kernel_stack:192kB pagetables:268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 21:17:44 2018] Node 0 Normal free:200640kB min:43740kB low:54672kB high:65608kB active_anon:343836kB inactive_anon:381644kB active_file:18869916kB inactive_file:18749108kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:528kB writeback:0kB mapped:149312kB shmem:551068kB slab_reclaimable:18460252kB slab_unreclaimable:5580340kB kernel_stack:4016kB pagetables:5104kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 1 Normal free:6840256kB min:45172kB low:56464kB high:67756kB active_anon:1993300kB inactive_anon:512392kB active_file:30007480kB inactive_file:5107396kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:548kB writeback:0kB mapped:144324kB shmem:352980kB slab_reclaimable:9710572kB slab_unreclaimable:7583196kB kernel_stack:10432kB pagetables:11396kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 21:17:44 2018] Node 0 DMA32: 2541*4kB (UEM) 3738*8kB (UEM) 1929*16kB (UEM) 3678*32kB (UEM) 1402*64kB (UEM) 167*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 300500kB [Sun Jul 1 21:17:44 2018] Node 0 Normal: 26871*4kB (UEM) 6078*8kB (UEM) 1281*16kB (UEM) 616*32kB (UEM) 90*64kB (UEM) 13*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 203996kB [Sun Jul 1 21:17:44 2018] Node 1 Normal: 227618*4kB (UEM) 234307*8kB (UEM) 239174*16kB (UEM) 7106*32kB (UEM) 73*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6843776kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] 18433225 total pagecache pages [Sun Jul 1 21:17:44 2018] 564 pages in swap cache [Sun Jul 1 21:17:44 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 21:17:44 2018] Free swap = 4180704kB [Sun Jul 1 21:17:44 2018] Total swap = 4194300kB [Sun Jul 1 21:17:44 2018] 33530455 pages RAM [Sun Jul 1 21:17:44 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 21:17:44 2018] 594386 pages reserved [Sun Jul 1 21:17:44 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 21:17:44 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 21:17:44 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 21:17:44 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 21:17:44 2018] Call Trace: [Sun Jul 1 21:17:44 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 21:17:44 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 21:17:44 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 21:17:44 2018] [] alloc_pages_current+0x98/0x110 [Sun Jul 1 21:17:44 2018] [] __get_free_pages+0xe/0x40 [Sun Jul 1 21:17:44 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Sun Jul 1 21:17:44 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Sun Jul 1 21:17:44 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 21:17:44 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 21:17:44 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 21:17:44 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 21:17:44 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 21:17:44 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 21:17:44 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 21:17:44 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 21:17:44 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 21:17:44 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 21:17:44 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] Mem-Info: [Sun Jul 1 21:17:44 2018] active_anon:584597 inactive_anon:225747 isolated_anon:0 active_file:12229783 inactive_file:5974029 isolated_file:2 unevictable:17363 dirty:280 writeback:0 unstable:0 slab_reclaimable:7216416 slab_unreclaimable:3439641 mapped:73724 shmem:226598 pagetables:4192 bounce:0 free:1838514 free_pcp:33 free_cma:0 [Sun Jul 1 21:17:44 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 21:17:44 2018] Node 0 DMA32 free:300400kB min:1184kB low:1480kB high:1776kB active_anon:1252kB inactive_anon:8952kB active_file:41736kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):8kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:44kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694840kB slab_unreclaimable:594964kB kernel_stack:192kB pagetables:268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 21:17:44 2018] Node 0 Normal free:200640kB min:43740kB low:54672kB high:65608kB active_anon:343836kB inactive_anon:381644kB active_file:18869916kB inactive_file:18749108kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:528kB writeback:0kB mapped:149312kB shmem:551068kB slab_reclaimable:18460252kB slab_unreclaimable:5580340kB kernel_stack:4016kB pagetables:5104kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 1 Normal free:6840256kB min:45172kB low:56464kB high:67756kB active_anon:1993300kB inactive_anon:512392kB active_file:30007480kB inactive_file:5107396kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:548kB writeback:0kB mapped:144324kB shmem:352980kB slab_reclaimable:9710572kB slab_unreclaimable:7583196kB kernel_stack:10432kB pagetables:11396kB unstable:0kB bounce:0kB free_pcp:316kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 21:17:44 2018] Node 0 DMA32: 2541*4kB (UEM) 3738*8kB (UEM) 1929*16kB (UEM) 3678*32kB (UEM) 1402*64kB (UEM) 167*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 300500kB [Sun Jul 1 21:17:44 2018] Node 0 Normal: 26871*4kB (UEM) 6079*8kB (UEM) 1281*16kB (UEM) 616*32kB (UEM) 90*64kB (UEM) 13*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 204004kB [Sun Jul 1 21:17:44 2018] Node 1 Normal: 227495*4kB (UEM) 234307*8kB (UEM) 239173*16kB (UEM) 7108*32kB (UEM) 73*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6843332kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] 18433225 total pagecache pages [Sun Jul 1 21:17:44 2018] 564 pages in swap cache [Sun Jul 1 21:17:44 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 21:17:44 2018] Free swap = 4180704kB [Sun Jul 1 21:17:44 2018] Total swap = 4194300kB [Sun Jul 1 21:17:44 2018] 33530455 pages RAM [Sun Jul 1 21:17:44 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 21:17:44 2018] 594386 pages reserved [Sun Jul 1 21:17:44 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 21:17:44 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 21:17:44 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 21:17:44 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 21:17:44 2018] Call Trace: [Sun Jul 1 21:17:44 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 21:17:44 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 21:17:44 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Sun Jul 1 21:17:44 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Sun Jul 1 21:17:44 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 21:17:44 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 21:17:44 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 21:17:44 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 21:17:44 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 21:17:44 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 21:17:44 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 21:17:44 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 21:17:44 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 21:17:44 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 21:17:44 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] Mem-Info: [Sun Jul 1 21:17:44 2018] active_anon:584597 inactive_anon:225747 isolated_anon:0 active_file:12229783 inactive_file:5974029 isolated_file:2 unevictable:17363 dirty:280 writeback:0 unstable:0 slab_reclaimable:7216416 slab_unreclaimable:3439641 mapped:73724 shmem:226598 pagetables:4192 bounce:0 free:1838371 free_pcp:240 free_cma:0 [Sun Jul 1 21:17:44 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 21:17:44 2018] Node 0 DMA32 free:300400kB min:1184kB low:1480kB high:1776kB active_anon:1252kB inactive_anon:8952kB active_file:41736kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):8kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:44kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694840kB slab_unreclaimable:594964kB kernel_stack:192kB pagetables:268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 21:17:44 2018] Node 0 Normal free:200640kB min:43740kB low:54672kB high:65608kB active_anon:343836kB inactive_anon:381644kB active_file:18869916kB inactive_file:18749108kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:528kB writeback:0kB mapped:149312kB shmem:551068kB slab_reclaimable:18460252kB slab_unreclaimable:5580340kB kernel_stack:4016kB pagetables:5104kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 1 Normal free:6839684kB min:45172kB low:56464kB high:67756kB active_anon:1993300kB inactive_anon:512392kB active_file:30007480kB inactive_file:5107396kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:548kB writeback:0kB mapped:144324kB shmem:352980kB slab_reclaimable:9710572kB slab_unreclaimable:7583196kB kernel_stack:10432kB pagetables:11396kB unstable:0kB bounce:0kB free_pcp:960kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 21:17:44 2018] Node 0 DMA32: 2541*4kB (UEM) 3738*8kB (UEM) 1929*16kB (UEM) 3678*32kB (UEM) 1402*64kB (UEM) 167*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 300500kB [Sun Jul 1 21:17:44 2018] Node 0 Normal: 26873*4kB (UEM) 6079*8kB (UEM) 1281*16kB (UEM) 616*32kB (UEM) 90*64kB (UEM) 13*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 204012kB [Sun Jul 1 21:17:44 2018] Node 1 Normal: 227344*4kB (UEM) 234306*8kB (UEM) 239172*16kB (UEM) 7109*32kB (UEM) 73*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6842736kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] 18433225 total pagecache pages [Sun Jul 1 21:17:44 2018] 564 pages in swap cache [Sun Jul 1 21:17:44 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 21:17:44 2018] Free swap = 4180704kB [Sun Jul 1 21:17:44 2018] Total swap = 4194300kB [Sun Jul 1 21:17:44 2018] 33530455 pages RAM [Sun Jul 1 21:17:44 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 21:17:44 2018] 594386 pages reserved [Sun Jul 1 21:17:44 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 21:17:44 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 21:17:44 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 21:17:44 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 21:17:44 2018] Call Trace: [Sun Jul 1 21:17:44 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 21:17:44 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 21:17:44 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 21:17:44 2018] [] alloc_pages_current+0x98/0x110 [Sun Jul 1 21:17:44 2018] [] __get_free_pages+0xe/0x40 [Sun Jul 1 21:17:44 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Sun Jul 1 21:17:44 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Sun Jul 1 21:17:44 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 21:17:44 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 21:17:44 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 21:17:44 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 21:17:44 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 21:17:44 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 21:17:44 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 21:17:44 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 21:17:44 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 21:17:44 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 21:17:44 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] Mem-Info: [Sun Jul 1 21:17:44 2018] active_anon:584597 inactive_anon:225747 isolated_anon:0 active_file:12229783 inactive_file:5974029 isolated_file:2 unevictable:17363 dirty:280 writeback:0 unstable:0 slab_reclaimable:7216416 slab_unreclaimable:3439641 mapped:73724 shmem:226598 pagetables:4192 bounce:0 free:1838489 free_pcp:30 free_cma:0 [Sun Jul 1 21:17:44 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 21:17:44 2018] Node 0 DMA32 free:300400kB min:1184kB low:1480kB high:1776kB active_anon:1252kB inactive_anon:8952kB active_file:41736kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):8kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:44kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694840kB slab_unreclaimable:594964kB kernel_stack:192kB pagetables:268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 21:17:44 2018] Node 0 Normal free:200640kB min:43740kB low:54672kB high:65608kB active_anon:343836kB inactive_anon:381644kB active_file:18869916kB inactive_file:18749108kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:528kB writeback:0kB mapped:149312kB shmem:551068kB slab_reclaimable:18460252kB slab_unreclaimable:5580340kB kernel_stack:4016kB pagetables:5104kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 1 Normal free:6840156kB min:45172kB low:56464kB high:67756kB active_anon:1993300kB inactive_anon:512392kB active_file:30007480kB inactive_file:5107396kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:548kB writeback:0kB mapped:144324kB shmem:352980kB slab_reclaimable:9710572kB slab_unreclaimable:7583196kB kernel_stack:10432kB pagetables:11396kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 21:17:44 2018] Node 0 DMA32: 2541*4kB (UEM) 3738*8kB (UEM) 1929*16kB (UEM) 3678*32kB (UEM) 1402*64kB (UEM) 167*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 300500kB [Sun Jul 1 21:17:44 2018] Node 0 Normal: 26873*4kB (UEM) 6079*8kB (UEM) 1281*16kB (UEM) 616*32kB (UEM) 90*64kB (UEM) 13*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 204012kB [Sun Jul 1 21:17:44 2018] Node 1 Normal: 227583*4kB (UEM) 234306*8kB (UEM) 239172*16kB (UEM) 7109*32kB (UEM) 73*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6843692kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] 18433225 total pagecache pages [Sun Jul 1 21:17:44 2018] 564 pages in swap cache [Sun Jul 1 21:17:44 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 21:17:44 2018] Free swap = 4180704kB [Sun Jul 1 21:17:44 2018] Total swap = 4194300kB [Sun Jul 1 21:17:44 2018] 33530455 pages RAM [Sun Jul 1 21:17:44 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 21:17:44 2018] 594386 pages reserved [Sun Jul 1 21:17:44 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 21:17:44 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 21:17:44 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 21:17:44 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 21:17:44 2018] Call Trace: [Sun Jul 1 21:17:44 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 21:17:44 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 21:17:44 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 21:17:44 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Sun Jul 1 21:17:44 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Sun Jul 1 21:17:44 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 21:17:44 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 21:17:44 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 21:17:44 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 21:17:44 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 21:17:44 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 21:17:44 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 21:17:44 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 21:17:44 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 21:17:44 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 21:17:44 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] Mem-Info: [Sun Jul 1 21:17:44 2018] active_anon:584597 inactive_anon:225747 isolated_anon:0 active_file:12229783 inactive_file:5974029 isolated_file:2 unevictable:17363 dirty:280 writeback:0 unstable:0 slab_reclaimable:7216416 slab_unreclaimable:3439641 mapped:73724 shmem:226598 pagetables:4192 bounce:0 free:1838489 free_pcp:88 free_cma:0 [Sun Jul 1 21:17:44 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 21:17:44 2018] Node 0 DMA32 free:300400kB min:1184kB low:1480kB high:1776kB active_anon:1252kB inactive_anon:8952kB active_file:41736kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):8kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:44kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694840kB slab_unreclaimable:594964kB kernel_stack:192kB pagetables:268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 21:17:44 2018] Node 0 Normal free:200640kB min:43740kB low:54672kB high:65608kB active_anon:343836kB inactive_anon:381644kB active_file:18869916kB inactive_file:18749108kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:528kB writeback:0kB mapped:149312kB shmem:551068kB slab_reclaimable:18460252kB slab_unreclaimable:5580340kB kernel_stack:4016kB pagetables:5104kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 1 Normal free:6840156kB min:45172kB low:56464kB high:67756kB active_anon:1993772kB inactive_anon:512392kB active_file:30007480kB inactive_file:5107396kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:548kB writeback:0kB mapped:144324kB shmem:352980kB slab_reclaimable:9710572kB slab_unreclaimable:7583196kB kernel_stack:10432kB pagetables:11396kB unstable:0kB bounce:0kB free_pcp:308kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 21:17:44 2018] Node 0 DMA32: 2541*4kB (UEM) 3738*8kB (UEM) 1929*16kB (UEM) 3678*32kB (UEM) 1402*64kB (UEM) 167*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 300500kB [Sun Jul 1 21:17:44 2018] Node 0 Normal: 26874*4kB (UEM) 6079*8kB (UEM) 1281*16kB (UEM) 616*32kB (UEM) 90*64kB (UEM) 13*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 204016kB [Sun Jul 1 21:17:44 2018] Node 1 Normal: 227388*4kB (UEM) 234306*8kB (UEM) 239174*16kB (UEM) 7109*32kB (UEM) 73*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6842944kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] 18433225 total pagecache pages [Sun Jul 1 21:17:44 2018] 564 pages in swap cache [Sun Jul 1 21:17:44 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 21:17:44 2018] Free swap = 4180704kB [Sun Jul 1 21:17:44 2018] Total swap = 4194300kB [Sun Jul 1 21:17:44 2018] 33530455 pages RAM [Sun Jul 1 21:17:44 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 21:17:44 2018] 594386 pages reserved [Sun Jul 1 21:17:44 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 21:17:44 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 21:17:44 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 21:17:44 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 21:17:44 2018] Call Trace: [Sun Jul 1 21:17:44 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 21:17:44 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 21:17:44 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 21:17:44 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 21:17:44 2018] [] alloc_pages_current+0x98/0x110 [Sun Jul 1 21:17:44 2018] [] __get_free_pages+0xe/0x40 [Sun Jul 1 21:17:44 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Sun Jul 1 21:17:44 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Sun Jul 1 21:17:44 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 21:17:44 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 21:17:44 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 21:17:44 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 21:17:44 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 21:17:44 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 21:17:44 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 21:17:44 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 21:17:44 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 21:17:44 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 21:17:44 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 21:17:44 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 21:17:44 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 21:17:44 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 21:17:44 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 21:17:44 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 21:17:44 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 21:17:44 2018] Mem-Info: [Sun Jul 1 21:17:44 2018] active_anon:584597 inactive_anon:225747 isolated_anon:0 active_file:12229783 inactive_file:5974029 isolated_file:2 unevictable:17363 dirty:280 writeback:0 unstable:0 slab_reclaimable:7216416 slab_unreclaimable:3439641 mapped:73724 shmem:226598 pagetables:4192 bounce:0 free:1838355 free_pcp:132 free_cma:0 [Sun Jul 1 21:17:44 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 21:17:44 2018] Node 0 DMA32 free:300400kB min:1184kB low:1480kB high:1776kB active_anon:1252kB inactive_anon:8952kB active_file:41736kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):8kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:44kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694840kB slab_unreclaimable:594964kB kernel_stack:192kB pagetables:268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 21:17:44 2018] Node 0 Normal free:200640kB min:43740kB low:54672kB high:65608kB active_anon:343836kB inactive_anon:381644kB active_file:18869916kB inactive_file:18749108kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:528kB writeback:0kB mapped:149312kB shmem:551068kB slab_reclaimable:18460252kB slab_unreclaimable:5580340kB kernel_stack:4016kB pagetables:5104kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 1 Normal free:6839620kB min:45172kB low:56464kB high:67756kB active_anon:1993300kB inactive_anon:512392kB active_file:30007480kB inactive_file:5107396kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:548kB writeback:0kB mapped:144324kB shmem:352980kB slab_reclaimable:9710572kB slab_unreclaimable:7583196kB kernel_stack:10432kB pagetables:11396kB unstable:0kB bounce:0kB free_pcp:488kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 21:17:44 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 21:17:44 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 21:17:44 2018] Node 0 DMA32: 2541*4kB (UEM) 3738*8kB (UEM) 1929*16kB (UEM) 3678*32kB (UEM) 1402*64kB (UEM) 167*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 300500kB [Sun Jul 1 21:17:44 2018] Node 0 Normal: 26878*4kB (UEM) 6079*8kB (UEM) 1281*16kB (UEM) 616*32kB (UEM) 90*64kB (UEM) 13*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 204032kB [Sun Jul 1 21:17:44 2018] Node 1 Normal: 227444*4kB (UEM) 234306*8kB (UEM) 239175*16kB (UEM) 7109*32kB (UEM) 73*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6843184kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 21:17:44 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 21:17:44 2018] 18433225 total pagecache pages [Sun Jul 1 21:17:44 2018] 564 pages in swap cache [Sun Jul 1 21:17:44 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 21:17:44 2018] Free swap = 4180704kB [Sun Jul 1 21:17:44 2018] Total swap = 4194300kB [Sun Jul 1 21:17:44 2018] 33530455 pages RAM [Sun Jul 1 21:17:44 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 21:17:44 2018] 594386 pages reserved [Sun Jul 1 21:17:44 2018] LNetError: 5577:0:(o2iblnd.c:934:kiblnd_create_conn()) Can't create QP: -12, send_wr: 409, recv_wr: 4, send_sge: 30, recv_sge: 1 [Sun Jul 1 21:17:44 2018] LNetError: 5577:0:(o2iblnd.c:934:kiblnd_create_conn()) Skipped 1 previous similar message [Sun Jul 1 21:18:42 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 21:18:42 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) Skipped 1776960 previous similar messages [Sun Jul 1 21:18:42 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 21:18:42 2018] Lustre: Skipped 1776960 previous similar messages [Sun Jul 1 21:18:42 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 21:18:42 2018] Lustre: Skipped 1776960 previous similar messages [Sun Jul 1 21:29:57 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 21:29:57 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) Skipped 1300995 previous similar messages [Sun Jul 1 21:29:57 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 21:29:57 2018] Lustre: Skipped 1300995 previous similar messages [Sun Jul 1 21:29:57 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 21:29:57 2018] Lustre: Skipped 1300995 previous similar messages [Sun Jul 1 21:40:22 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 21:40:22 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) Skipped 594688 previous similar messages [Sun Jul 1 21:40:22 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 21:40:22 2018] Lustre: Skipped 594688 previous similar messages [Sun Jul 1 21:40:22 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 21:40:22 2018] Lustre: Skipped 594688 previous similar messages [Sun Jul 1 21:50:22 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 21:50:22 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) Skipped 951162 previous similar messages [Sun Jul 1 21:50:22 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 21:50:22 2018] Lustre: Skipped 951163 previous similar messages [Sun Jul 1 21:50:22 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 21:50:22 2018] Lustre: Skipped 951163 previous similar messages [Sun Jul 1 22:00:22 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 22:00:22 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) Skipped 2399486 previous similar messages [Sun Jul 1 22:00:22 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 22:00:22 2018] Lustre: Skipped 2399486 previous similar messages [Sun Jul 1 22:00:22 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 22:00:22 2018] Lustre: Skipped 2399486 previous similar messages [Sun Jul 1 22:09:04 2018] warn_alloc_failed: 22 callbacks suppressed [Sun Jul 1 22:09:04 2018] kworker/11:2: page allocation failure: order:8, mode:0x80d0 [Sun Jul 1 22:09:04 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 22:09:04 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 22:09:04 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 22:09:04 2018] Call Trace: [Sun Jul 1 22:09:04 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 22:09:04 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 22:09:04 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Sun Jul 1 22:09:04 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Sun Jul 1 22:09:04 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 22:09:04 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 22:09:04 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 22:09:04 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 22:09:04 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 22:09:04 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 22:09:04 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 22:09:04 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 22:09:04 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 22:09:04 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 22:09:04 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] Mem-Info: [Sun Jul 1 22:09:04 2018] active_anon:586045 inactive_anon:227794 isolated_anon:0 active_file:12240160 inactive_file:5974011 isolated_file:0 unevictable:17363 dirty:23 writeback:0 unstable:0 slab_reclaimable:7216303 slab_unreclaimable:3447842 mapped:72664 shmem:228645 pagetables:4256 bounce:0 free:1816519 free_pcp:0 free_cma:0 [Sun Jul 1 22:09:04 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 22:09:04 2018] Node 0 DMA32 free:297336kB min:1184kB low:1480kB high:1776kB active_anon:1884kB inactive_anon:9068kB active_file:43456kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694880kB slab_unreclaimable:595320kB kernel_stack:176kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 22:09:04 2018] Node 0 Normal free:153868kB min:43740kB low:54672kB high:65608kB active_anon:351196kB inactive_anon:382316kB active_file:18889712kB inactive_file:18749076kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:12kB writeback:0kB mapped:144124kB shmem:551068kB slab_reclaimable:18460544kB slab_unreclaimable:5598516kB kernel_stack:3936kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 1 Normal free:6802112kB min:45172kB low:56464kB high:67756kB active_anon:1991100kB inactive_anon:519792kB active_file:30027472kB inactive_file:5107356kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:80kB writeback:0kB mapped:145272kB shmem:361168kB slab_reclaimable:9709788kB slab_unreclaimable:7597468kB kernel_stack:10480kB pagetables:11536kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 22:09:04 2018] Node 0 DMA32: 2412*4kB (UEM) 3528*8kB (UEM) 1798*16kB (UEM) 3700*32kB (UEM) 1404*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 297424kB [Sun Jul 1 22:09:04 2018] Node 0 Normal: 22983*4kB (UEM) 4468*8kB (UEM) 1345*16kB (EM) 243*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 156972kB [Sun Jul 1 22:09:04 2018] Node 1 Normal: 227391*4kB (UEM) 234622*8kB (UEM) 239683*16kB (UEM) 5769*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6806076kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] 18445639 total pagecache pages [Sun Jul 1 22:09:04 2018] 564 pages in swap cache [Sun Jul 1 22:09:04 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 22:09:04 2018] Free swap = 4180704kB [Sun Jul 1 22:09:04 2018] Total swap = 4194300kB [Sun Jul 1 22:09:04 2018] 33530455 pages RAM [Sun Jul 1 22:09:04 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 22:09:04 2018] 594386 pages reserved [Sun Jul 1 22:09:04 2018] kworker/11:2: page allocation failure: order:8, mode:0x80d0 [Sun Jul 1 22:09:04 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 22:09:04 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 22:09:04 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 22:09:04 2018] Call Trace: [Sun Jul 1 22:09:04 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 22:09:04 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 22:09:04 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 22:09:04 2018] [] alloc_pages_current+0x98/0x110 [Sun Jul 1 22:09:04 2018] [] __get_free_pages+0xe/0x40 [Sun Jul 1 22:09:04 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Sun Jul 1 22:09:04 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Sun Jul 1 22:09:04 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 22:09:04 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 22:09:04 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 22:09:04 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 22:09:04 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 22:09:04 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 22:09:04 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 22:09:04 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 22:09:04 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 22:09:04 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 22:09:04 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] Mem-Info: [Sun Jul 1 22:09:04 2018] active_anon:586045 inactive_anon:227794 isolated_anon:0 active_file:12240160 inactive_file:5974011 isolated_file:0 unevictable:17363 dirty:23 writeback:0 unstable:0 slab_reclaimable:7216303 slab_unreclaimable:3447842 mapped:72664 shmem:228645 pagetables:4256 bounce:0 free:1816519 free_pcp:1 free_cma:0 [Sun Jul 1 22:09:04 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 22:09:04 2018] Node 0 DMA32 free:297336kB min:1184kB low:1480kB high:1776kB active_anon:1884kB inactive_anon:9068kB active_file:43456kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694880kB slab_unreclaimable:595320kB kernel_stack:176kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 22:09:04 2018] Node 0 Normal free:153868kB min:43740kB low:54672kB high:65608kB active_anon:351196kB inactive_anon:382316kB active_file:18889712kB inactive_file:18749076kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:12kB writeback:0kB mapped:144124kB shmem:551068kB slab_reclaimable:18460544kB slab_unreclaimable:5598516kB kernel_stack:3936kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 1 Normal free:6802112kB min:45172kB low:56464kB high:67756kB active_anon:1991100kB inactive_anon:519792kB active_file:30027472kB inactive_file:5107356kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:80kB writeback:0kB mapped:145272kB shmem:361168kB slab_reclaimable:9709788kB slab_unreclaimable:7597468kB kernel_stack:10480kB pagetables:11536kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 22:09:04 2018] Node 0 DMA32: 2412*4kB (UEM) 3528*8kB (UEM) 1798*16kB (UEM) 3700*32kB (UEM) 1404*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 297424kB [Sun Jul 1 22:09:04 2018] Node 0 Normal: 22983*4kB (UEM) 4468*8kB (UEM) 1345*16kB (EM) 243*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 156972kB [Sun Jul 1 22:09:04 2018] Node 1 Normal: 227391*4kB (UEM) 234622*8kB (UEM) 239685*16kB (UEM) 5769*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6806108kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] 18445639 total pagecache pages [Sun Jul 1 22:09:04 2018] 564 pages in swap cache [Sun Jul 1 22:09:04 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 22:09:04 2018] Free swap = 4180704kB [Sun Jul 1 22:09:04 2018] Total swap = 4194300kB [Sun Jul 1 22:09:04 2018] 33530455 pages RAM [Sun Jul 1 22:09:04 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 22:09:04 2018] 594386 pages reserved [Sun Jul 1 22:09:04 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 22:09:04 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 22:09:04 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 22:09:04 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 22:09:04 2018] Call Trace: [Sun Jul 1 22:09:04 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 22:09:04 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 22:09:04 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 22:09:04 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Sun Jul 1 22:09:04 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Sun Jul 1 22:09:04 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 22:09:04 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 22:09:04 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 22:09:04 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 22:09:04 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 22:09:04 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 22:09:04 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 22:09:04 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 22:09:04 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 22:09:04 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 22:09:04 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] Mem-Info: [Sun Jul 1 22:09:04 2018] active_anon:586045 inactive_anon:227794 isolated_anon:0 active_file:12240160 inactive_file:5974011 isolated_file:0 unevictable:17363 dirty:23 writeback:0 unstable:0 slab_reclaimable:7216303 slab_unreclaimable:3447842 mapped:72664 shmem:228645 pagetables:4256 bounce:0 free:1816519 free_pcp:1 free_cma:0 [Sun Jul 1 22:09:04 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 22:09:04 2018] Node 0 DMA32 free:297336kB min:1184kB low:1480kB high:1776kB active_anon:1884kB inactive_anon:9068kB active_file:43456kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694880kB slab_unreclaimable:595320kB kernel_stack:176kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 22:09:04 2018] Node 0 Normal free:153868kB min:43740kB low:54672kB high:65608kB active_anon:351196kB inactive_anon:382316kB active_file:18889712kB inactive_file:18749076kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:12kB writeback:0kB mapped:144124kB shmem:551068kB slab_reclaimable:18460544kB slab_unreclaimable:5598516kB kernel_stack:3936kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 1 Normal free:6802112kB min:45172kB low:56464kB high:67756kB active_anon:1991100kB inactive_anon:519792kB active_file:30027472kB inactive_file:5107356kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:80kB writeback:0kB mapped:145272kB shmem:361168kB slab_reclaimable:9709788kB slab_unreclaimable:7597468kB kernel_stack:10480kB pagetables:11536kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 22:09:04 2018] Node 0 DMA32: 2412*4kB (UEM) 3528*8kB (UEM) 1798*16kB (UEM) 3700*32kB (UEM) 1404*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 297424kB [Sun Jul 1 22:09:04 2018] Node 0 Normal: 22986*4kB (UEM) 4468*8kB (UEM) 1346*16kB (EM) 243*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 157000kB [Sun Jul 1 22:09:04 2018] Node 1 Normal: 227392*4kB (UEM) 234623*8kB (UEM) 239685*16kB (UEM) 5769*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6806120kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] 18445639 total pagecache pages [Sun Jul 1 22:09:04 2018] 564 pages in swap cache [Sun Jul 1 22:09:04 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 22:09:04 2018] Free swap = 4180704kB [Sun Jul 1 22:09:04 2018] Total swap = 4194300kB [Sun Jul 1 22:09:04 2018] 33530455 pages RAM [Sun Jul 1 22:09:04 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 22:09:04 2018] 594386 pages reserved [Sun Jul 1 22:09:04 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 22:09:04 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 22:09:04 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 22:09:04 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 22:09:04 2018] Call Trace: [Sun Jul 1 22:09:04 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 22:09:04 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 22:09:04 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 22:09:04 2018] [] alloc_pages_current+0x98/0x110 [Sun Jul 1 22:09:04 2018] [] __get_free_pages+0xe/0x40 [Sun Jul 1 22:09:04 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Sun Jul 1 22:09:04 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Sun Jul 1 22:09:04 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 22:09:04 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 22:09:04 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 22:09:04 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 22:09:04 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 22:09:04 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 22:09:04 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 22:09:04 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 22:09:04 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 22:09:04 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 22:09:04 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] Mem-Info: [Sun Jul 1 22:09:04 2018] active_anon:586045 inactive_anon:227794 isolated_anon:0 active_file:12240160 inactive_file:5974011 isolated_file:0 unevictable:17363 dirty:23 writeback:0 unstable:0 slab_reclaimable:7216303 slab_unreclaimable:3447842 mapped:72664 shmem:228645 pagetables:4256 bounce:0 free:1816519 free_pcp:0 free_cma:0 [Sun Jul 1 22:09:04 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 22:09:04 2018] Node 0 DMA32 free:297336kB min:1184kB low:1480kB high:1776kB active_anon:1884kB inactive_anon:9068kB active_file:43456kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694880kB slab_unreclaimable:595320kB kernel_stack:176kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 22:09:04 2018] Node 0 Normal free:153868kB min:43740kB low:54672kB high:65608kB active_anon:351196kB inactive_anon:382316kB active_file:18889712kB inactive_file:18749076kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:12kB writeback:0kB mapped:144124kB shmem:551068kB slab_reclaimable:18460544kB slab_unreclaimable:5598516kB kernel_stack:3936kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 1 Normal free:6802112kB min:45172kB low:56464kB high:67756kB active_anon:1991100kB inactive_anon:519792kB active_file:30027472kB inactive_file:5107356kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:80kB writeback:0kB mapped:145272kB shmem:361168kB slab_reclaimable:9709788kB slab_unreclaimable:7597468kB kernel_stack:10480kB pagetables:11536kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 22:09:04 2018] Node 0 DMA32: 2412*4kB (UEM) 3528*8kB (UEM) 1798*16kB (UEM) 3700*32kB (UEM) 1404*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 297424kB [Sun Jul 1 22:09:04 2018] Node 0 Normal: 22989*4kB (UEM) 4468*8kB (UEM) 1346*16kB (EM) 243*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 157012kB [Sun Jul 1 22:09:04 2018] Node 1 Normal: 227391*4kB (UEM) 234623*8kB (UEM) 239684*16kB (UEM) 5769*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6806100kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] 18445639 total pagecache pages [Sun Jul 1 22:09:04 2018] 564 pages in swap cache [Sun Jul 1 22:09:04 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 22:09:04 2018] Free swap = 4180704kB [Sun Jul 1 22:09:04 2018] Total swap = 4194300kB [Sun Jul 1 22:09:04 2018] 33530455 pages RAM [Sun Jul 1 22:09:04 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 22:09:04 2018] 594386 pages reserved [Sun Jul 1 22:09:04 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 22:09:04 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 22:09:04 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 22:09:04 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 22:09:04 2018] Call Trace: [Sun Jul 1 22:09:04 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 22:09:04 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 22:09:04 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Sun Jul 1 22:09:04 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Sun Jul 1 22:09:04 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 22:09:04 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 22:09:04 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 22:09:04 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 22:09:04 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 22:09:04 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 22:09:04 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 22:09:04 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 22:09:04 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 22:09:04 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 22:09:04 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] Mem-Info: [Sun Jul 1 22:09:04 2018] active_anon:586163 inactive_anon:227794 isolated_anon:0 active_file:12240160 inactive_file:5974011 isolated_file:0 unevictable:17363 dirty:23 writeback:0 unstable:0 slab_reclaimable:7216303 slab_unreclaimable:3447842 mapped:72664 shmem:228645 pagetables:4256 bounce:0 free:1816519 free_pcp:7 free_cma:0 [Sun Jul 1 22:09:04 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 22:09:04 2018] Node 0 DMA32 free:297336kB min:1184kB low:1480kB high:1776kB active_anon:1884kB inactive_anon:9068kB active_file:43456kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694880kB slab_unreclaimable:595320kB kernel_stack:176kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 22:09:04 2018] Node 0 Normal free:153868kB min:43740kB low:54672kB high:65608kB active_anon:351196kB inactive_anon:382316kB active_file:18889712kB inactive_file:18749076kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:12kB writeback:0kB mapped:144124kB shmem:551068kB slab_reclaimable:18460544kB slab_unreclaimable:5598516kB kernel_stack:3936kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 1 Normal free:6802112kB min:45172kB low:56464kB high:67756kB active_anon:1991572kB inactive_anon:519792kB active_file:30027472kB inactive_file:5107356kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:80kB writeback:0kB mapped:145272kB shmem:361168kB slab_reclaimable:9709788kB slab_unreclaimable:7597468kB kernel_stack:10480kB pagetables:11536kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 22:09:04 2018] Node 0 DMA32: 2412*4kB (UEM) 3528*8kB (UEM) 1798*16kB (UEM) 3700*32kB (UEM) 1404*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 297424kB [Sun Jul 1 22:09:04 2018] Node 0 Normal: 22990*4kB (UEM) 4468*8kB (UEM) 1346*16kB (EM) 243*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 157016kB [Sun Jul 1 22:09:04 2018] Node 1 Normal: 227381*4kB (UEM) 234620*8kB (UEM) 239672*16kB (UE) 5757*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6805460kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] 18445639 total pagecache pages [Sun Jul 1 22:09:04 2018] 564 pages in swap cache [Sun Jul 1 22:09:04 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 22:09:04 2018] Free swap = 4180704kB [Sun Jul 1 22:09:04 2018] Total swap = 4194300kB [Sun Jul 1 22:09:04 2018] 33530455 pages RAM [Sun Jul 1 22:09:04 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 22:09:04 2018] 594386 pages reserved [Sun Jul 1 22:09:04 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 22:09:04 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 22:09:04 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 22:09:04 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 22:09:04 2018] Call Trace: [Sun Jul 1 22:09:04 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 22:09:04 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 22:09:04 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 22:09:04 2018] [] alloc_pages_current+0x98/0x110 [Sun Jul 1 22:09:04 2018] [] __get_free_pages+0xe/0x40 [Sun Jul 1 22:09:04 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Sun Jul 1 22:09:04 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Sun Jul 1 22:09:04 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 22:09:04 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 22:09:04 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 22:09:04 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 22:09:04 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 22:09:04 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 22:09:04 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 22:09:04 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 22:09:04 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 22:09:04 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 22:09:04 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] Mem-Info: [Sun Jul 1 22:09:04 2018] active_anon:586163 inactive_anon:227794 isolated_anon:0 active_file:12240160 inactive_file:5974011 isolated_file:0 unevictable:17363 dirty:23 writeback:0 unstable:0 slab_reclaimable:7216303 slab_unreclaimable:3447842 mapped:72664 shmem:228645 pagetables:4256 bounce:0 free:1816398 free_pcp:15 free_cma:0 [Sun Jul 1 22:09:04 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 22:09:04 2018] Node 0 DMA32 free:297336kB min:1184kB low:1480kB high:1776kB active_anon:1884kB inactive_anon:9068kB active_file:43456kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694880kB slab_unreclaimable:595320kB kernel_stack:176kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 22:09:04 2018] Node 0 Normal free:153868kB min:43740kB low:54672kB high:65608kB active_anon:351196kB inactive_anon:382316kB active_file:18889712kB inactive_file:18749076kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:12kB writeback:0kB mapped:144124kB shmem:551068kB slab_reclaimable:18460544kB slab_unreclaimable:5598516kB kernel_stack:3936kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 1 Normal free:6801628kB min:45172kB low:56464kB high:67756kB active_anon:1991572kB inactive_anon:519792kB active_file:30027472kB inactive_file:5107356kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:80kB writeback:0kB mapped:145272kB shmem:361168kB slab_reclaimable:9709788kB slab_unreclaimable:7597468kB kernel_stack:10480kB pagetables:11536kB unstable:0kB bounce:0kB free_pcp:176kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 22:09:04 2018] Node 0 DMA32: 2412*4kB (UEM) 3528*8kB (UEM) 1798*16kB (UEM) 3700*32kB (UEM) 1404*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 297424kB [Sun Jul 1 22:09:04 2018] Node 0 Normal: 22990*4kB (UEM) 4468*8kB (UEM) 1346*16kB (EM) 243*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 157016kB [Sun Jul 1 22:09:04 2018] Node 1 Normal: 227366*4kB (UE) 234622*8kB (UEM) 239674*16kB (UEM) 5752*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6805288kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] 18445639 total pagecache pages [Sun Jul 1 22:09:04 2018] 564 pages in swap cache [Sun Jul 1 22:09:04 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 22:09:04 2018] Free swap = 4180704kB [Sun Jul 1 22:09:04 2018] Total swap = 4194300kB [Sun Jul 1 22:09:04 2018] 33530455 pages RAM [Sun Jul 1 22:09:04 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 22:09:04 2018] 594386 pages reserved [Sun Jul 1 22:09:04 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 22:09:04 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 22:09:04 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 22:09:04 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 22:09:04 2018] Call Trace: [Sun Jul 1 22:09:04 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 22:09:04 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 22:09:04 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 22:09:04 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Sun Jul 1 22:09:04 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Sun Jul 1 22:09:04 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 22:09:04 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 22:09:04 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 22:09:04 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 22:09:04 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 22:09:04 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 22:09:04 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 22:09:04 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 22:09:04 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 22:09:04 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 22:09:04 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] Mem-Info: [Sun Jul 1 22:09:04 2018] active_anon:586045 inactive_anon:227794 isolated_anon:0 active_file:12240160 inactive_file:5974011 isolated_file:0 unevictable:17363 dirty:23 writeback:0 unstable:0 slab_reclaimable:7216303 slab_unreclaimable:3447842 mapped:72664 shmem:228645 pagetables:4256 bounce:0 free:1816516 free_pcp:30 free_cma:0 [Sun Jul 1 22:09:04 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 22:09:04 2018] Node 0 DMA32 free:297336kB min:1184kB low:1480kB high:1776kB active_anon:1884kB inactive_anon:9068kB active_file:43456kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694880kB slab_unreclaimable:595320kB kernel_stack:176kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 22:09:04 2018] Node 0 Normal free:153868kB min:43740kB low:54672kB high:65608kB active_anon:351196kB inactive_anon:382316kB active_file:18889712kB inactive_file:18749076kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:12kB writeback:0kB mapped:144124kB shmem:551068kB slab_reclaimable:18460544kB slab_unreclaimable:5598516kB kernel_stack:3936kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 1 Normal free:6802100kB min:45172kB low:56464kB high:67756kB active_anon:1991100kB inactive_anon:519792kB active_file:30027472kB inactive_file:5107356kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:80kB writeback:0kB mapped:145272kB shmem:361168kB slab_reclaimable:9709788kB slab_unreclaimable:7597468kB kernel_stack:10480kB pagetables:11536kB unstable:0kB bounce:0kB free_pcp:88kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 22:09:04 2018] Node 0 DMA32: 2412*4kB (UEM) 3528*8kB (UEM) 1798*16kB (UEM) 3700*32kB (UEM) 1404*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 297424kB [Sun Jul 1 22:09:04 2018] Node 0 Normal: 22991*4kB (UEM) 4468*8kB (UEM) 1346*16kB (EM) 243*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 157020kB [Sun Jul 1 22:09:04 2018] Node 1 Normal: 227388*4kB (UEM) 234623*8kB (UEM) 239682*16kB (UEM) 5756*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6805640kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] 18445639 total pagecache pages [Sun Jul 1 22:09:04 2018] 564 pages in swap cache [Sun Jul 1 22:09:04 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 22:09:04 2018] Free swap = 4180704kB [Sun Jul 1 22:09:04 2018] Total swap = 4194300kB [Sun Jul 1 22:09:04 2018] 33530455 pages RAM [Sun Jul 1 22:09:04 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 22:09:04 2018] 594386 pages reserved [Sun Jul 1 22:09:04 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 22:09:04 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 22:09:04 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 22:09:04 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 22:09:04 2018] Call Trace: [Sun Jul 1 22:09:04 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 22:09:04 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 22:09:04 2018] [] alloc_pages_current+0x98/0x110 [Sun Jul 1 22:09:04 2018] [] __get_free_pages+0xe/0x40 [Sun Jul 1 22:09:04 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Sun Jul 1 22:09:04 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Sun Jul 1 22:09:04 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 22:09:04 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 22:09:04 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 22:09:04 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 22:09:04 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 22:09:04 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 22:09:04 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 22:09:04 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 22:09:04 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 22:09:04 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 22:09:04 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] Mem-Info: [Sun Jul 1 22:09:04 2018] active_anon:586045 inactive_anon:227794 isolated_anon:0 active_file:12240160 inactive_file:5974011 isolated_file:0 unevictable:17363 dirty:23 writeback:0 unstable:0 slab_reclaimable:7216303 slab_unreclaimable:3447842 mapped:72664 shmem:228645 pagetables:4256 bounce:0 free:1816494 free_pcp:45 free_cma:0 [Sun Jul 1 22:09:04 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 22:09:04 2018] Node 0 DMA32 free:297336kB min:1184kB low:1480kB high:1776kB active_anon:1884kB inactive_anon:9068kB active_file:43456kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694880kB slab_unreclaimable:595320kB kernel_stack:176kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 22:09:04 2018] Node 0 Normal free:153868kB min:43740kB low:54672kB high:65608kB active_anon:351196kB inactive_anon:382316kB active_file:18889712kB inactive_file:18749076kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:12kB writeback:0kB mapped:144124kB shmem:551068kB slab_reclaimable:18460544kB slab_unreclaimable:5598516kB kernel_stack:3936kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 1 Normal free:6802012kB min:45172kB low:56464kB high:67756kB active_anon:1991100kB inactive_anon:519792kB active_file:30027472kB inactive_file:5107356kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:80kB writeback:0kB mapped:145272kB shmem:361168kB slab_reclaimable:9709788kB slab_unreclaimable:7597468kB kernel_stack:10480kB pagetables:11536kB unstable:0kB bounce:0kB free_pcp:148kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 22:09:04 2018] Node 0 DMA32: 2412*4kB (UEM) 3528*8kB (UEM) 1798*16kB (UEM) 3700*32kB (UEM) 1404*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 297424kB [Sun Jul 1 22:09:04 2018] Node 0 Normal: 22993*4kB (UEM) 4468*8kB (UEM) 1346*16kB (EM) 243*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 157028kB [Sun Jul 1 22:09:04 2018] Node 1 Normal: 227357*4kB (UE) 234622*8kB (UE) 239684*16kB (UE) 5755*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6805508kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] 18445639 total pagecache pages [Sun Jul 1 22:09:04 2018] 564 pages in swap cache [Sun Jul 1 22:09:04 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 22:09:04 2018] Free swap = 4180704kB [Sun Jul 1 22:09:04 2018] Total swap = 4194300kB [Sun Jul 1 22:09:04 2018] 33530455 pages RAM [Sun Jul 1 22:09:04 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 22:09:04 2018] 594386 pages reserved [Sun Jul 1 22:09:04 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 22:09:04 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 22:09:04 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 22:09:04 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 22:09:04 2018] Call Trace: [Sun Jul 1 22:09:04 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 22:09:04 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 22:09:04 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 22:09:04 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Sun Jul 1 22:09:04 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Sun Jul 1 22:09:04 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 22:09:04 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 22:09:04 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 22:09:04 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 22:09:04 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 22:09:04 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 22:09:04 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 22:09:04 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 22:09:04 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 22:09:04 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 22:09:04 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] Mem-Info: [Sun Jul 1 22:09:04 2018] active_anon:586163 inactive_anon:227794 isolated_anon:0 active_file:12240160 inactive_file:5974011 isolated_file:0 unevictable:17363 dirty:23 writeback:0 unstable:0 slab_reclaimable:7216303 slab_unreclaimable:3447842 mapped:72664 shmem:228645 pagetables:4256 bounce:0 free:1816366 free_pcp:5 free_cma:0 [Sun Jul 1 22:09:04 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 22:09:04 2018] Node 0 DMA32 free:297336kB min:1184kB low:1480kB high:1776kB active_anon:1884kB inactive_anon:9068kB active_file:43456kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694880kB slab_unreclaimable:595320kB kernel_stack:176kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 22:09:04 2018] Node 0 Normal free:153868kB min:43740kB low:54672kB high:65608kB active_anon:351196kB inactive_anon:382316kB active_file:18889712kB inactive_file:18749076kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:12kB writeback:0kB mapped:144124kB shmem:551068kB slab_reclaimable:18460544kB slab_unreclaimable:5598516kB kernel_stack:3936kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 1 Normal free:6801500kB min:45172kB low:56464kB high:67756kB active_anon:1991572kB inactive_anon:519792kB active_file:30027472kB inactive_file:5107356kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:80kB writeback:0kB mapped:145272kB shmem:361168kB slab_reclaimable:9709788kB slab_unreclaimable:7597468kB kernel_stack:10480kB pagetables:11536kB unstable:0kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 22:09:04 2018] Node 0 DMA32: 2412*4kB (UEM) 3528*8kB (UEM) 1798*16kB (UEM) 3700*32kB (UEM) 1404*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 297424kB [Sun Jul 1 22:09:04 2018] Node 0 Normal: 22993*4kB (UEM) 4468*8kB (UEM) 1346*16kB (EM) 243*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 157028kB [Sun Jul 1 22:09:04 2018] Node 1 Normal: 227372*4kB (UEM) 234624*8kB (UEM) 239688*16kB (UEM) 5746*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6805360kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] 18445639 total pagecache pages [Sun Jul 1 22:09:04 2018] 564 pages in swap cache [Sun Jul 1 22:09:04 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 22:09:04 2018] Free swap = 4180704kB [Sun Jul 1 22:09:04 2018] Total swap = 4194300kB [Sun Jul 1 22:09:04 2018] 33530455 pages RAM [Sun Jul 1 22:09:04 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 22:09:04 2018] 594386 pages reserved [Sun Jul 1 22:09:04 2018] kworker/11:2: page allocation failure: order:9, mode:0x80d0 [Sun Jul 1 22:09:04 2018] CPU: 11 PID: 5577 Comm: kworker/11:2 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Sun Jul 1 22:09:04 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Sun Jul 1 22:09:04 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Sun Jul 1 22:09:04 2018] Call Trace: [Sun Jul 1 22:09:04 2018] [] dump_stack+0x19/0x1b [Sun Jul 1 22:09:04 2018] [] warn_alloc_failed+0x110/0x180 [Sun Jul 1 22:09:04 2018] [] ? drain_pages+0xb0/0xb0 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Sun Jul 1 22:09:04 2018] [] __alloc_pages_nodemask+0x405/0x420 [Sun Jul 1 22:09:04 2018] [] alloc_pages_current+0x98/0x110 [Sun Jul 1 22:09:04 2018] [] __get_free_pages+0xe/0x40 [Sun Jul 1 22:09:04 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Sun Jul 1 22:09:04 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Sun Jul 1 22:09:04 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? debugfs_create_file+0x1f/0x30 [Sun Jul 1 22:09:04 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Sun Jul 1 22:09:04 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Sun Jul 1 22:09:04 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Sun Jul 1 22:09:04 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Sun Jul 1 22:09:04 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Sun Jul 1 22:09:04 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Sun Jul 1 22:09:04 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Sun Jul 1 22:09:04 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Sun Jul 1 22:09:04 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Sun Jul 1 22:09:04 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Sun Jul 1 22:09:04 2018] [] ? __schedule+0x424/0x9b0 [Sun Jul 1 22:09:04 2018] [] process_one_work+0x17a/0x440 [Sun Jul 1 22:09:04 2018] [] worker_thread+0x126/0x3c0 [Sun Jul 1 22:09:04 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Sun Jul 1 22:09:04 2018] [] kthread+0xcf/0xe0 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] [] ret_from_fork+0x58/0x90 [Sun Jul 1 22:09:04 2018] [] ? insert_kthread_work+0x40/0x40 [Sun Jul 1 22:09:04 2018] Mem-Info: [Sun Jul 1 22:09:04 2018] active_anon:586045 inactive_anon:227794 isolated_anon:0 active_file:12240160 inactive_file:5974011 isolated_file:0 unevictable:17363 dirty:23 writeback:0 unstable:0 slab_reclaimable:7216303 slab_unreclaimable:3447842 mapped:72664 shmem:228645 pagetables:4256 bounce:0 free:1816484 free_pcp:30 free_cma:0 [Sun Jul 1 22:09:04 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 1690 64141 64141 [Sun Jul 1 22:09:04 2018] Node 0 DMA32 free:297336kB min:1184kB low:1480kB high:1776kB active_anon:1884kB inactive_anon:9068kB active_file:43456kB inactive_file:39612kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:694880kB slab_unreclaimable:595320kB kernel_stack:176kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 62450 62450 [Sun Jul 1 22:09:04 2018] Node 0 Normal free:153868kB min:43740kB low:54672kB high:65608kB active_anon:351196kB inactive_anon:382316kB active_file:18889712kB inactive_file:18749076kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:12kB writeback:0kB mapped:144124kB shmem:551068kB slab_reclaimable:18460544kB slab_unreclaimable:5598516kB kernel_stack:3936kB pagetables:5252kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 1 Normal free:6801972kB min:45172kB low:56464kB high:67756kB active_anon:1991100kB inactive_anon:519792kB active_file:30027472kB inactive_file:5107356kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:80kB writeback:0kB mapped:145272kB shmem:361168kB slab_reclaimable:9709788kB slab_unreclaimable:7597468kB kernel_stack:10480kB pagetables:11536kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Sun Jul 1 22:09:04 2018] lowmem_reserve[]: 0 0 0 0 [Sun Jul 1 22:09:04 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Sun Jul 1 22:09:04 2018] Node 0 DMA32: 2412*4kB (UEM) 3528*8kB (UEM) 1798*16kB (UEM) 3700*32kB (UEM) 1404*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 297424kB [Sun Jul 1 22:09:04 2018] Node 0 Normal: 22966*4kB (UEM) 4468*8kB (UEM) 1346*16kB (EM) 243*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 156920kB [Sun Jul 1 22:09:04 2018] Node 1 Normal: 227390*4kB (UEM) 234621*8kB (UE) 239687*16kB (UEM) 5772*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6806224kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Sun Jul 1 22:09:04 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sun Jul 1 22:09:04 2018] 18445639 total pagecache pages [Sun Jul 1 22:09:04 2018] 564 pages in swap cache [Sun Jul 1 22:09:04 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Sun Jul 1 22:09:04 2018] Free swap = 4180704kB [Sun Jul 1 22:09:04 2018] Total swap = 4194300kB [Sun Jul 1 22:09:04 2018] 33530455 pages RAM [Sun Jul 1 22:09:04 2018] 0 pages HighMem/MovableOnly [Sun Jul 1 22:09:04 2018] 594386 pages reserved [Sun Jul 1 22:09:04 2018] LNet: 5577:0:(o2iblnd.c:943:kiblnd_create_conn()) peer 172.16.229.39@o2ib - queue depth reduced from 8 to 1 to allow for qp creation [Sun Jul 1 22:11:11 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 22:11:11 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) Skipped 2157471 previous similar messages [Sun Jul 1 22:11:11 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 22:11:11 2018] Lustre: Skipped 2157470 previous similar messages [Sun Jul 1 22:11:11 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 22:11:11 2018] Lustre: Skipped 2157470 previous similar messages [Sun Jul 1 22:22:08 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 22:22:08 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) Skipped 2204931 previous similar messages [Sun Jul 1 22:22:08 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 22:22:08 2018] Lustre: Skipped 2204931 previous similar messages [Sun Jul 1 22:22:08 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 22:22:08 2018] Lustre: Skipped 2204931 previous similar messages [Sun Jul 1 22:32:08 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 22:32:08 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) Skipped 1925352 previous similar messages [Sun Jul 1 22:32:09 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 22:32:09 2018] Lustre: Skipped 1925352 previous similar messages [Sun Jul 1 22:32:09 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 22:32:09 2018] Lustre: Skipped 1925352 previous similar messages [Sun Jul 1 22:42:09 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 22:42:09 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) Skipped 2419146 previous similar messages [Sun Jul 1 22:42:09 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 22:42:09 2018] Lustre: Skipped 2419234 previous similar messages [Sun Jul 1 22:42:09 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 22:42:09 2018] Lustre: Skipped 2419234 previous similar messages [Sun Jul 1 22:53:25 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 22:53:25 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) Skipped 2317273 previous similar messages [Sun Jul 1 22:53:25 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 22:53:25 2018] Lustre: Skipped 2317185 previous similar messages [Sun Jul 1 22:53:25 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 22:53:25 2018] Lustre: Skipped 2317185 previous similar messages [Sun Jul 1 23:03:25 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 23:03:25 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) Skipped 1943893 previous similar messages [Sun Jul 1 23:03:25 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 23:03:25 2018] Lustre: Skipped 1943996 previous similar messages [Sun Jul 1 23:03:25 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 23:03:25 2018] Lustre: Skipped 1943996 previous similar messages [Sun Jul 1 23:13:25 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 23:13:25 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) Skipped 1439753 previous similar messages [Sun Jul 1 23:13:25 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 23:13:25 2018] Lustre: Skipped 1439691 previous similar messages [Sun Jul 1 23:13:25 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 23:13:25 2018] Lustre: Skipped 1439691 previous similar messages [Sun Jul 1 23:24:55 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 23:24:55 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) Skipped 1360753 previous similar messages [Sun Jul 1 23:24:55 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 23:24:55 2018] Lustre: Skipped 1360712 previous similar messages [Sun Jul 1 23:24:55 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 23:24:55 2018] Lustre: Skipped 1360712 previous similar messages [Sun Jul 1 23:35:51 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 23:35:51 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) Skipped 2657637 previous similar messages [Sun Jul 1 23:35:51 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 23:35:51 2018] Lustre: Skipped 2657637 previous similar messages [Sun Jul 1 23:35:51 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 23:35:51 2018] Lustre: Skipped 2657637 previous similar messages [Sun Jul 1 23:46:51 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 23:46:51 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) Skipped 1740903 previous similar messages [Sun Jul 1 23:46:51 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 23:46:51 2018] Lustre: Skipped 1740903 previous similar messages [Sun Jul 1 23:46:51 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 23:46:51 2018] Lustre: Skipped 1740903 previous similar messages [Sun Jul 1 23:56:51 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Sun Jul 1 23:56:51 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) Skipped 1926526 previous similar messages [Sun Jul 1 23:56:51 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Sun Jul 1 23:56:51 2018] Lustre: Skipped 1926565 previous similar messages [Sun Jul 1 23:56:51 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Sun Jul 1 23:56:51 2018] Lustre: Skipped 1926565 previous similar messages [Mon Jul 2 00:07:41 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 00:07:41 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) Skipped 1714332 previous similar messages [Mon Jul 2 00:07:41 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 00:07:41 2018] Lustre: Skipped 1714293 previous similar messages [Mon Jul 2 00:07:41 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 00:07:41 2018] Lustre: Skipped 1714293 previous similar messages [Mon Jul 2 00:19:19 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 00:19:19 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 1939781 previous similar messages [Mon Jul 2 00:19:19 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 00:19:19 2018] Lustre: Skipped 1939781 previous similar messages [Mon Jul 2 00:19:19 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 00:19:19 2018] Lustre: Skipped 1939781 previous similar messages [Mon Jul 2 00:29:19 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 00:29:19 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) Skipped 2374604 previous similar messages [Mon Jul 2 00:29:19 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 00:29:19 2018] Lustre: Skipped 2374604 previous similar messages [Mon Jul 2 00:29:19 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 00:29:19 2018] Lustre: Skipped 2374604 previous similar messages [Mon Jul 2 00:39:56 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 00:39:56 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) Skipped 2087761 previous similar messages [Mon Jul 2 00:39:56 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 00:39:56 2018] Lustre: Skipped 2087761 previous similar messages [Mon Jul 2 00:39:56 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 00:39:56 2018] Lustre: Skipped 2087761 previous similar messages [Mon Jul 2 00:49:56 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 00:49:56 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) Skipped 2397650 previous similar messages [Mon Jul 2 00:49:56 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 00:49:56 2018] Lustre: Skipped 2397651 previous similar messages [Mon Jul 2 00:49:56 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 00:49:56 2018] Lustre: Skipped 2397651 previous similar messages [Mon Jul 2 01:00:38 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 01:00:38 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 2089668 previous similar messages [Mon Jul 2 01:00:38 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 01:00:38 2018] Lustre: Skipped 2089667 previous similar messages [Mon Jul 2 01:00:38 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 01:00:38 2018] Lustre: Skipped 2089667 previous similar messages [Mon Jul 2 01:10:38 2018] LustreError: 146455:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 01:10:38 2018] LustreError: 146455:0:(lod_dev.c:1414:lod_sync()) Skipped 2476790 previous similar messages [Mon Jul 2 01:10:38 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 01:10:38 2018] Lustre: Skipped 2476791 previous similar messages [Mon Jul 2 01:10:38 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 01:10:38 2018] Lustre: Skipped 2476791 previous similar messages [Mon Jul 2 01:20:38 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 01:20:38 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) Skipped 1968140 previous similar messages [Mon Jul 2 01:20:38 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 01:20:38 2018] Lustre: Skipped 1968140 previous similar messages [Mon Jul 2 01:20:38 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 01:20:38 2018] Lustre: Skipped 1968140 previous similar messages [Mon Jul 2 01:30:38 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 01:30:38 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) Skipped 1439855 previous similar messages [Mon Jul 2 01:30:38 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 01:30:38 2018] Lustre: Skipped 1439891 previous similar messages [Mon Jul 2 01:30:38 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 01:30:38 2018] Lustre: Skipped 1439891 previous similar messages [Mon Jul 2 01:40:38 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 01:40:38 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) Skipped 2366008 previous similar messages [Mon Jul 2 01:40:38 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 01:40:38 2018] Lustre: Skipped 2365972 previous similar messages [Mon Jul 2 01:40:38 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 01:40:38 2018] Lustre: Skipped 2365972 previous similar messages [Mon Jul 2 01:50:38 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 01:50:38 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) Skipped 1914512 previous similar messages [Mon Jul 2 01:50:38 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 01:50:38 2018] Lustre: Skipped 1914639 previous similar messages [Mon Jul 2 01:50:38 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 01:50:38 2018] Lustre: Skipped 1914639 previous similar messages [Mon Jul 2 02:00:38 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 02:00:38 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) Skipped 1922016 previous similar messages [Mon Jul 2 02:00:38 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 02:00:38 2018] Lustre: Skipped 1922014 previous similar messages [Mon Jul 2 02:00:38 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 02:00:38 2018] Lustre: Skipped 1922014 previous similar messages [Mon Jul 2 02:10:38 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 02:10:38 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) Skipped 2436445 previous similar messages [Mon Jul 2 02:10:38 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 02:10:38 2018] Lustre: Skipped 2436450 previous similar messages [Mon Jul 2 02:10:38 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 02:10:38 2018] Lustre: Skipped 2436450 previous similar messages [Mon Jul 2 02:21:46 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 02:21:46 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) Skipped 2287503 previous similar messages [Mon Jul 2 02:21:46 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 02:21:46 2018] Lustre: Skipped 2287372 previous similar messages [Mon Jul 2 02:21:46 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 02:21:46 2018] Lustre: Skipped 2287372 previous similar messages [Mon Jul 2 02:31:46 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 02:31:46 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) Skipped 2386905 previous similar messages [Mon Jul 2 02:31:46 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 02:31:46 2018] Lustre: Skipped 2386905 previous similar messages [Mon Jul 2 02:31:46 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 02:31:46 2018] Lustre: Skipped 2386905 previous similar messages [Mon Jul 2 02:41:46 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 02:41:46 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) Skipped 2414118 previous similar messages [Mon Jul 2 02:41:46 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 02:41:46 2018] Lustre: Skipped 2414128 previous similar messages [Mon Jul 2 02:41:46 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 02:41:46 2018] Lustre: Skipped 2414128 previous similar messages [Mon Jul 2 02:52:34 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 02:52:34 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) Skipped 2199506 previous similar messages [Mon Jul 2 02:52:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 02:52:34 2018] Lustre: Skipped 2199496 previous similar messages [Mon Jul 2 02:52:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 02:52:34 2018] Lustre: Skipped 2199496 previous similar messages [Mon Jul 2 03:02:34 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 03:02:34 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 1943451 previous similar messages [Mon Jul 2 03:02:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 03:02:34 2018] Lustre: Skipped 1943492 previous similar messages [Mon Jul 2 03:02:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 03:02:34 2018] Lustre: Skipped 1943492 previous similar messages [Mon Jul 2 03:12:34 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 03:12:34 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) Skipped 1920684 previous similar messages [Mon Jul 2 03:12:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 03:12:34 2018] Lustre: Skipped 1920643 previous similar messages [Mon Jul 2 03:12:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 03:12:34 2018] Lustre: Skipped 1920643 previous similar messages [Mon Jul 2 03:22:34 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 03:22:34 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) Skipped 1434994 previous similar messages [Mon Jul 2 03:22:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 03:22:34 2018] Lustre: Skipped 1435072 previous similar messages [Mon Jul 2 03:22:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 03:22:34 2018] Lustre: Skipped 1435072 previous similar messages [Mon Jul 2 03:32:34 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 03:32:34 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 1888614 previous similar messages [Mon Jul 2 03:32:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 03:32:34 2018] Lustre: Skipped 1888537 previous similar messages [Mon Jul 2 03:32:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 03:32:34 2018] Lustre: Skipped 1888537 previous similar messages [Mon Jul 2 03:42:34 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 03:42:34 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) Skipped 2533520 previous similar messages [Mon Jul 2 03:42:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 03:42:35 2018] Lustre: Skipped 2533605 previous similar messages [Mon Jul 2 03:42:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 03:42:35 2018] Lustre: Skipped 2533605 previous similar messages [Mon Jul 2 03:52:34 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 03:52:34 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) Skipped 2418033 previous similar messages [Mon Jul 2 03:52:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 03:52:35 2018] Lustre: Skipped 2417947 previous similar messages [Mon Jul 2 03:52:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 03:52:35 2018] Lustre: Skipped 2417947 previous similar messages [Mon Jul 2 04:02:35 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 04:02:35 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) Skipped 1899729 previous similar messages [Mon Jul 2 04:02:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 04:02:35 2018] Lustre: Skipped 1899967 previous similar messages [Mon Jul 2 04:02:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 04:02:35 2018] Lustre: Skipped 1899967 previous similar messages [Mon Jul 2 04:12:35 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 04:12:35 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) Skipped 2440858 previous similar messages [Mon Jul 2 04:12:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 04:12:35 2018] Lustre: Skipped 2440866 previous similar messages [Mon Jul 2 04:12:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 04:12:35 2018] Lustre: Skipped 2440866 previous similar messages [Mon Jul 2 04:22:35 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 04:22:35 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 2405609 previous similar messages [Mon Jul 2 04:22:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 04:22:35 2018] Lustre: Skipped 2405608 previous similar messages [Mon Jul 2 04:22:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 04:22:35 2018] Lustre: Skipped 2405608 previous similar messages [Mon Jul 2 04:32:35 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 04:32:35 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) Skipped 1915253 previous similar messages [Mon Jul 2 04:32:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 04:32:35 2018] Lustre: Skipped 1915249 previous similar messages [Mon Jul 2 04:32:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 04:32:35 2018] Lustre: Skipped 1915249 previous similar messages [Mon Jul 2 04:42:35 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 04:42:35 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) Skipped 2384402 previous similar messages [Mon Jul 2 04:42:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 04:42:35 2018] Lustre: Skipped 2384381 previous similar messages [Mon Jul 2 04:42:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 04:42:35 2018] Lustre: Skipped 2384381 previous similar messages [Mon Jul 2 04:52:35 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 04:52:35 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) Skipped 2478142 previous similar messages [Mon Jul 2 04:52:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 04:52:35 2018] Lustre: Skipped 2478157 previous similar messages [Mon Jul 2 04:52:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 04:52:35 2018] Lustre: Skipped 2478157 previous similar messages [Mon Jul 2 05:02:35 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 05:02:35 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) Skipped 1952866 previous similar messages [Mon Jul 2 05:02:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 05:02:35 2018] Lustre: Skipped 1952920 previous similar messages [Mon Jul 2 05:02:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 05:02:35 2018] Lustre: Skipped 1952920 previous similar messages [Mon Jul 2 05:12:35 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 05:12:35 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2639179 previous similar messages [Mon Jul 2 05:12:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 05:12:35 2018] Lustre: Skipped 2639827 previous similar messages [Mon Jul 2 05:12:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 05:12:35 2018] Lustre: Skipped 2639828 previous similar messages [Mon Jul 2 05:22:35 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 05:22:35 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) Skipped 6384781 previous similar messages [Mon Jul 2 05:22:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 05:22:35 2018] Lustre: Skipped 6384706 previous similar messages [Mon Jul 2 05:22:35 2018] Lustre: lustre-MDT0000: Client 39fee214-4fae-7d31-5802-27d145d78e9d (at 172.16.229.20@o2ib) reconnecting [Mon Jul 2 05:22:35 2018] Lustre: Skipped 6384708 previous similar messages [Mon Jul 2 05:32:35 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 05:32:35 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) Skipped 6507994 previous similar messages [Mon Jul 2 05:32:35 2018] Lustre: lustre-MDT0000: Client 39fee214-4fae-7d31-5802-27d145d78e9d (at 172.16.229.20@o2ib) reconnecting [Mon Jul 2 05:32:35 2018] Lustre: Skipped 6508070 previous similar messages [Mon Jul 2 05:32:35 2018] Lustre: lustre-MDT0000: Connection restored to 1db370a5-2ae6-d677-ed6e-2bfba780742d (at 172.16.229.20@o2ib) [Mon Jul 2 05:32:35 2018] Lustre: Skipped 6508073 previous similar messages [Mon Jul 2 05:35:47 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880103cb6c00 x1601781515387424/t0(0) o37->39fee214-4fae-7d31-5802-27d145d78e9d@172.16.229.20@o2ib:213/0 lens 568/440 e 0 to 0 dl 1530473548 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 05:35:47 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88011caeda00 x1601781515387424/t0(0) o37->39fee214-4fae-7d31-5802-27d145d78e9d@172.16.229.20@o2ib:213/0 lens 568/440 e 0 to 0 dl 1530473548 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 05:35:47 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 10 previous similar messages [Mon Jul 2 05:35:47 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.20@o2ib [Mon Jul 2 05:35:47 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 2 previous similar messages [Mon Jul 2 05:35:47 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 13096 previous similar messages [Mon Jul 2 05:37:13 2018] LNet: 156322:0:(o2iblnd_cb.c:2502:kiblnd_passive_connect()) Conn stale 172.16.229.20@o2ib version 12/12 incarnation 1527577546426277/1530473628691982 [Mon Jul 2 05:37:13 2018] warn_alloc_failed: 4 callbacks suppressed [Mon Jul 2 05:37:13 2018] kworker/11:1: page allocation failure: order:8, mode:0x80d0 [Mon Jul 2 05:37:13 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 05:37:13 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 05:37:13 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 05:37:13 2018] Call Trace: [Mon Jul 2 05:37:13 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 05:37:13 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 05:37:13 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 05:37:13 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 05:37:13 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 05:37:13 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 05:37:13 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 05:37:13 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 05:37:13 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 05:37:13 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 05:37:13 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 05:37:13 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 05:37:13 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 05:37:13 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 05:37:13 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] Mem-Info: [Mon Jul 2 05:37:13 2018] active_anon:591203 inactive_anon:227791 isolated_anon:0 active_file:12233305 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7217152 slab_unreclaimable:3453716 mapped:74342 shmem:228645 pagetables:4195 bounce:0 free:1818806 free_pcp:0 free_cma:0 [Mon Jul 2 05:37:13 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 05:37:13 2018] Node 0 DMA32 free:283368kB min:1184kB low:1480kB high:1776kB active_anon:3260kB inactive_anon:9096kB active_file:50492kB inactive_file:39380kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:700312kB slab_unreclaimable:595564kB kernel_stack:272kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 05:37:13 2018] Node 0 Normal free:247292kB min:43740kB low:54672kB high:65608kB active_anon:337016kB inactive_anon:374140kB active_file:18808868kB inactive_file:18712188kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:143564kB shmem:551072kB slab_reclaimable:18480132kB slab_unreclaimable:5613364kB kernel_stack:4080kB pagetables:4988kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 1 Normal free:6731804kB min:45172kB low:56464kB high:67756kB active_anon:2024536kB inactive_anon:527928kB active_file:30073860kB inactive_file:5101500kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:36kB writeback:0kB mapped:152544kB shmem:361164kB slab_reclaimable:9688164kB slab_unreclaimable:7605872kB kernel_stack:10208kB pagetables:11480kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 05:37:13 2018] Node 0 DMA32: 2400*4kB (UEM) 2886*8kB (UEM) 1462*16kB (UEM) 3620*32kB (UEM) 1391*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 283472kB [Mon Jul 2 05:37:13 2018] Node 0 Normal: 37789*4kB (UEM) 5533*8kB (UEM) 2789*16kB (UEM) 304*32kB (UEM) 36*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 252076kB [Mon Jul 2 05:37:13 2018] Node 1 Normal: 234413*4kB (UEM) 240844*8kB (UEM) 241916*16kB (UEM) 28*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6735956kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] 18428034 total pagecache pages [Mon Jul 2 05:37:13 2018] 564 pages in swap cache [Mon Jul 2 05:37:13 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 05:37:13 2018] Free swap = 4180704kB [Mon Jul 2 05:37:13 2018] Total swap = 4194300kB [Mon Jul 2 05:37:13 2018] 33530455 pages RAM [Mon Jul 2 05:37:13 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 05:37:13 2018] 594386 pages reserved [Mon Jul 2 05:37:13 2018] kworker/11:1: page allocation failure: order:8, mode:0x80d0 [Mon Jul 2 05:37:13 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 05:37:13 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 05:37:13 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 05:37:13 2018] Call Trace: [Mon Jul 2 05:37:13 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 05:37:13 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 05:37:13 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 05:37:13 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 05:37:13 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 05:37:13 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 05:37:13 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 05:37:13 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 05:37:13 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 05:37:13 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 05:37:13 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 05:37:13 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 05:37:13 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 05:37:13 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 05:37:13 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 05:37:13 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 05:37:13 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 05:37:13 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] Mem-Info: [Mon Jul 2 05:37:13 2018] active_anon:591203 inactive_anon:227791 isolated_anon:0 active_file:12233305 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7217152 slab_unreclaimable:3453716 mapped:74342 shmem:228645 pagetables:4195 bounce:0 free:1818806 free_pcp:2 free_cma:0 [Mon Jul 2 05:37:13 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 05:37:13 2018] Node 0 DMA32 free:283368kB min:1184kB low:1480kB high:1776kB active_anon:3260kB inactive_anon:9096kB active_file:50492kB inactive_file:39380kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:700312kB slab_unreclaimable:595564kB kernel_stack:272kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 05:37:13 2018] Node 0 Normal free:247292kB min:43740kB low:54672kB high:65608kB active_anon:337016kB inactive_anon:374140kB active_file:18808868kB inactive_file:18712188kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:143564kB shmem:551072kB slab_reclaimable:18480132kB slab_unreclaimable:5613364kB kernel_stack:4080kB pagetables:4988kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 1 Normal free:6731804kB min:45172kB low:56464kB high:67756kB active_anon:2024536kB inactive_anon:527928kB active_file:30073860kB inactive_file:5101500kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:36kB writeback:0kB mapped:152544kB shmem:361164kB slab_reclaimable:9688164kB slab_unreclaimable:7605872kB kernel_stack:10208kB pagetables:11480kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 05:37:13 2018] Node 0 DMA32: 2400*4kB (UEM) 2886*8kB (UEM) 1462*16kB (UEM) 3620*32kB (UEM) 1391*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 283472kB [Mon Jul 2 05:37:13 2018] Node 0 Normal: 37789*4kB (UEM) 5535*8kB (UEM) 2789*16kB (UEM) 304*32kB (UEM) 36*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 252092kB [Mon Jul 2 05:37:13 2018] Node 1 Normal: 234413*4kB (UEM) 240844*8kB (UEM) 241916*16kB (UEM) 28*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6735956kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] 18428034 total pagecache pages [Mon Jul 2 05:37:13 2018] 564 pages in swap cache [Mon Jul 2 05:37:13 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 05:37:13 2018] Free swap = 4180704kB [Mon Jul 2 05:37:13 2018] Total swap = 4194300kB [Mon Jul 2 05:37:13 2018] 33530455 pages RAM [Mon Jul 2 05:37:13 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 05:37:13 2018] 594386 pages reserved [Mon Jul 2 05:37:13 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 05:37:13 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 05:37:13 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 05:37:13 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 05:37:13 2018] Call Trace: [Mon Jul 2 05:37:13 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 05:37:13 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 05:37:13 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 05:37:13 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 05:37:13 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 05:37:13 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 05:37:13 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 05:37:13 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 05:37:13 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 05:37:13 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 05:37:13 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 05:37:13 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 05:37:13 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 05:37:13 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 05:37:13 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 05:37:13 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] Mem-Info: [Mon Jul 2 05:37:13 2018] active_anon:591203 inactive_anon:227791 isolated_anon:0 active_file:12233305 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7217152 slab_unreclaimable:3453716 mapped:74342 shmem:228645 pagetables:4195 bounce:0 free:1818806 free_pcp:5 free_cma:0 [Mon Jul 2 05:37:13 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 05:37:13 2018] Node 0 DMA32 free:283368kB min:1184kB low:1480kB high:1776kB active_anon:3260kB inactive_anon:9096kB active_file:50492kB inactive_file:39380kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:700312kB slab_unreclaimable:595564kB kernel_stack:272kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 05:37:13 2018] Node 0 Normal free:247292kB min:43740kB low:54672kB high:65608kB active_anon:337016kB inactive_anon:374140kB active_file:18808868kB inactive_file:18712188kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:143564kB shmem:551072kB slab_reclaimable:18480132kB slab_unreclaimable:5613364kB kernel_stack:4080kB pagetables:4988kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 1 Normal free:6731804kB min:45172kB low:56464kB high:67756kB active_anon:2024536kB inactive_anon:527928kB active_file:30073860kB inactive_file:5101500kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:36kB writeback:0kB mapped:152544kB shmem:361164kB slab_reclaimable:9688164kB slab_unreclaimable:7605872kB kernel_stack:10208kB pagetables:11480kB unstable:0kB bounce:0kB free_pcp:20kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 05:37:13 2018] Node 0 DMA32: 2400*4kB (UEM) 2886*8kB (UEM) 1462*16kB (UEM) 3620*32kB (UEM) 1391*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 283472kB [Mon Jul 2 05:37:13 2018] Node 0 Normal: 37789*4kB (UEM) 5535*8kB (UEM) 2789*16kB (UEM) 304*32kB (UEM) 36*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 252092kB [Mon Jul 2 05:37:13 2018] Node 1 Normal: 234395*4kB (UEM) 240841*8kB (UEM) 241916*16kB (UEM) 29*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6735892kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] 18428034 total pagecache pages [Mon Jul 2 05:37:13 2018] 564 pages in swap cache [Mon Jul 2 05:37:13 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 05:37:13 2018] Free swap = 4180704kB [Mon Jul 2 05:37:13 2018] Total swap = 4194300kB [Mon Jul 2 05:37:13 2018] 33530455 pages RAM [Mon Jul 2 05:37:13 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 05:37:13 2018] 594386 pages reserved [Mon Jul 2 05:37:13 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 05:37:13 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 05:37:13 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 05:37:13 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 05:37:13 2018] Call Trace: [Mon Jul 2 05:37:13 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 05:37:13 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 05:37:13 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 05:37:13 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 05:37:13 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 05:37:13 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 05:37:13 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 05:37:13 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 05:37:13 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 05:37:13 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 05:37:13 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 05:37:13 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 05:37:13 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 05:37:13 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 05:37:13 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 05:37:13 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 05:37:13 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 05:37:13 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] Mem-Info: [Mon Jul 2 05:37:13 2018] active_anon:591203 inactive_anon:227791 isolated_anon:0 active_file:12233305 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7217152 slab_unreclaimable:3453716 mapped:74342 shmem:228645 pagetables:4195 bounce:0 free:1818806 free_pcp:41 free_cma:0 [Mon Jul 2 05:37:13 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 05:37:13 2018] Node 0 DMA32 free:283368kB min:1184kB low:1480kB high:1776kB active_anon:3260kB inactive_anon:9096kB active_file:50492kB inactive_file:39380kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:700312kB slab_unreclaimable:595564kB kernel_stack:272kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 05:37:13 2018] Node 0 Normal free:247292kB min:43740kB low:54672kB high:65608kB active_anon:337016kB inactive_anon:374140kB active_file:18808868kB inactive_file:18712188kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:143564kB shmem:551072kB slab_reclaimable:18480132kB slab_unreclaimable:5613364kB kernel_stack:4080kB pagetables:4988kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 1 Normal free:6731804kB min:45172kB low:56464kB high:67756kB active_anon:2024536kB inactive_anon:527928kB active_file:30073860kB inactive_file:5101500kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:36kB writeback:0kB mapped:152544kB shmem:361164kB slab_reclaimable:9688164kB slab_unreclaimable:7605872kB kernel_stack:10208kB pagetables:11480kB unstable:0kB bounce:0kB free_pcp:128kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 05:37:13 2018] Node 0 DMA32: 2400*4kB (UEM) 2886*8kB (UEM) 1462*16kB (UEM) 3620*32kB (UEM) 1391*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 283472kB [Mon Jul 2 05:37:13 2018] Node 0 Normal: 37800*4kB (UEM) 5535*8kB (UEM) 2789*16kB (UEM) 304*32kB (UEM) 36*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 252136kB [Mon Jul 2 05:37:13 2018] Node 1 Normal: 234362*4kB (UEM) 240841*8kB (UE) 241893*16kB (UEM) 29*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6735392kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] 18428034 total pagecache pages [Mon Jul 2 05:37:13 2018] 564 pages in swap cache [Mon Jul 2 05:37:13 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 05:37:13 2018] Free swap = 4180704kB [Mon Jul 2 05:37:13 2018] Total swap = 4194300kB [Mon Jul 2 05:37:13 2018] 33530455 pages RAM [Mon Jul 2 05:37:13 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 05:37:13 2018] 594386 pages reserved [Mon Jul 2 05:37:13 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 05:37:13 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 05:37:13 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 05:37:13 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 05:37:13 2018] Call Trace: [Mon Jul 2 05:37:13 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 05:37:13 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 05:37:13 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 05:37:13 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 05:37:13 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 05:37:13 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 05:37:13 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 05:37:13 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 05:37:13 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 05:37:13 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 05:37:13 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 05:37:13 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 05:37:13 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 05:37:13 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 05:37:13 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 05:37:13 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] Mem-Info: [Mon Jul 2 05:37:13 2018] active_anon:591321 inactive_anon:227791 isolated_anon:0 active_file:12233305 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7217152 slab_unreclaimable:3453716 mapped:74342 shmem:228645 pagetables:4195 bounce:0 free:1818679 free_pcp:13 free_cma:0 [Mon Jul 2 05:37:13 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 05:37:13 2018] Node 0 DMA32 free:283368kB min:1184kB low:1480kB high:1776kB active_anon:3260kB inactive_anon:9096kB active_file:50492kB inactive_file:39380kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:700312kB slab_unreclaimable:595564kB kernel_stack:272kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 05:37:13 2018] Node 0 Normal free:247292kB min:43740kB low:54672kB high:65608kB active_anon:337016kB inactive_anon:374140kB active_file:18808868kB inactive_file:18712188kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:143564kB shmem:551072kB slab_reclaimable:18480132kB slab_unreclaimable:5613364kB kernel_stack:4080kB pagetables:4988kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 1 Normal free:6731296kB min:45172kB low:56464kB high:67756kB active_anon:2025008kB inactive_anon:527928kB active_file:30073860kB inactive_file:5101500kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:36kB writeback:0kB mapped:152544kB shmem:361164kB slab_reclaimable:9688164kB slab_unreclaimable:7605872kB kernel_stack:10208kB pagetables:11480kB unstable:0kB bounce:0kB free_pcp:24kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 05:37:13 2018] Node 0 DMA32: 2400*4kB (UEM) 2886*8kB (UEM) 1462*16kB (UEM) 3620*32kB (UEM) 1391*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 283472kB [Mon Jul 2 05:37:13 2018] Node 0 Normal: 37802*4kB (UEM) 5535*8kB (UEM) 2789*16kB (UEM) 304*32kB (UEM) 36*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 252144kB [Mon Jul 2 05:37:13 2018] Node 1 Normal: 234362*4kB (UE) 240842*8kB (UEM) 241888*16kB (UE) 22*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6735096kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] 18428034 total pagecache pages [Mon Jul 2 05:37:13 2018] 564 pages in swap cache [Mon Jul 2 05:37:13 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 05:37:13 2018] Free swap = 4180704kB [Mon Jul 2 05:37:13 2018] Total swap = 4194300kB [Mon Jul 2 05:37:13 2018] 33530455 pages RAM [Mon Jul 2 05:37:13 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 05:37:13 2018] 594386 pages reserved [Mon Jul 2 05:37:13 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 05:37:13 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 05:37:13 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 05:37:13 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 05:37:13 2018] Call Trace: [Mon Jul 2 05:37:13 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 05:37:13 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 05:37:13 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 05:37:13 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 05:37:13 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 05:37:13 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 05:37:13 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 05:37:13 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 05:37:13 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 05:37:13 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 05:37:13 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 05:37:13 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 05:37:13 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 05:37:13 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 05:37:13 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 05:37:13 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 05:37:13 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] Mem-Info: [Mon Jul 2 05:37:13 2018] active_anon:591321 inactive_anon:227791 isolated_anon:0 active_file:12233305 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7217152 slab_unreclaimable:3453716 mapped:74342 shmem:228645 pagetables:4195 bounce:0 free:1818679 free_pcp:4 free_cma:0 [Mon Jul 2 05:37:13 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 05:37:13 2018] Node 0 DMA32 free:283368kB min:1184kB low:1480kB high:1776kB active_anon:3260kB inactive_anon:9096kB active_file:50492kB inactive_file:39380kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:700312kB slab_unreclaimable:595564kB kernel_stack:272kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 05:37:13 2018] Node 0 Normal free:247292kB min:43740kB low:54672kB high:65608kB active_anon:337016kB inactive_anon:374140kB active_file:18808868kB inactive_file:18712188kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:143564kB shmem:551072kB slab_reclaimable:18480132kB slab_unreclaimable:5613364kB kernel_stack:4080kB pagetables:4988kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 1 Normal free:6731296kB min:45172kB low:56464kB high:67756kB active_anon:2025008kB inactive_anon:527928kB active_file:30073860kB inactive_file:5101500kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:36kB writeback:0kB mapped:152544kB shmem:361164kB slab_reclaimable:9688164kB slab_unreclaimable:7605872kB kernel_stack:10208kB pagetables:11480kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 05:37:13 2018] Node 0 DMA32: 2400*4kB (UEM) 2886*8kB (UEM) 1462*16kB (UEM) 3620*32kB (UEM) 1391*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 283472kB [Mon Jul 2 05:37:13 2018] Node 0 Normal: 37805*4kB (UEM) 5535*8kB (UEM) 2789*16kB (UEM) 304*32kB (UEM) 36*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 252156kB [Mon Jul 2 05:37:13 2018] Node 1 Normal: 234401*4kB (UE) 240847*8kB (UEM) 241919*16kB (UE) 5*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6735244kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] 18428034 total pagecache pages [Mon Jul 2 05:37:13 2018] 564 pages in swap cache [Mon Jul 2 05:37:13 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 05:37:13 2018] Free swap = 4180704kB [Mon Jul 2 05:37:13 2018] Total swap = 4194300kB [Mon Jul 2 05:37:13 2018] 33530455 pages RAM [Mon Jul 2 05:37:13 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 05:37:13 2018] 594386 pages reserved [Mon Jul 2 05:37:13 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 05:37:13 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 05:37:13 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 05:37:13 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 05:37:13 2018] Call Trace: [Mon Jul 2 05:37:13 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 05:37:13 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 05:37:13 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 05:37:13 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 05:37:13 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 05:37:13 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 05:37:13 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 05:37:13 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 05:37:13 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 05:37:13 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 05:37:13 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 05:37:13 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 05:37:13 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 05:37:13 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 05:37:13 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 05:37:13 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] Mem-Info: [Mon Jul 2 05:37:13 2018] active_anon:591321 inactive_anon:227791 isolated_anon:0 active_file:12233305 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7217152 slab_unreclaimable:3453716 mapped:74342 shmem:228645 pagetables:4195 bounce:0 free:1818679 free_pcp:162 free_cma:0 [Mon Jul 2 05:37:13 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 05:37:13 2018] Node 0 DMA32 free:283368kB min:1184kB low:1480kB high:1776kB active_anon:3260kB inactive_anon:9096kB active_file:50492kB inactive_file:39380kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:700312kB slab_unreclaimable:595564kB kernel_stack:272kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 05:37:13 2018] Node 0 Normal free:247292kB min:43740kB low:54672kB high:65608kB active_anon:337016kB inactive_anon:374140kB active_file:18808868kB inactive_file:18712188kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:143564kB shmem:551072kB slab_reclaimable:18480132kB slab_unreclaimable:5613364kB kernel_stack:4080kB pagetables:4988kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 1 Normal free:6731296kB min:45172kB low:56464kB high:67756kB active_anon:2025008kB inactive_anon:527928kB active_file:30073860kB inactive_file:5101500kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:36kB writeback:0kB mapped:152544kB shmem:361164kB slab_reclaimable:9688164kB slab_unreclaimable:7605872kB kernel_stack:10208kB pagetables:11480kB unstable:0kB bounce:0kB free_pcp:628kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 05:37:13 2018] Node 0 DMA32: 2400*4kB (UEM) 2886*8kB (UEM) 1462*16kB (UEM) 3620*32kB (UEM) 1391*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 283472kB [Mon Jul 2 05:37:13 2018] Node 0 Normal: 37805*4kB (UEM) 5535*8kB (UEM) 2789*16kB (UEM) 304*32kB (UEM) 36*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 252156kB [Mon Jul 2 05:37:13 2018] Node 1 Normal: 234409*4kB (UE) 240851*8kB (UE) 241917*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6735116kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] 18428034 total pagecache pages [Mon Jul 2 05:37:13 2018] 564 pages in swap cache [Mon Jul 2 05:37:13 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 05:37:13 2018] Free swap = 4180704kB [Mon Jul 2 05:37:13 2018] Total swap = 4194300kB [Mon Jul 2 05:37:13 2018] 33530455 pages RAM [Mon Jul 2 05:37:13 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 05:37:13 2018] 594386 pages reserved [Mon Jul 2 05:37:13 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 05:37:13 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 05:37:13 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 05:37:13 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 05:37:13 2018] Call Trace: [Mon Jul 2 05:37:13 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 05:37:13 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 05:37:13 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 05:37:13 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 05:37:13 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 05:37:13 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 05:37:13 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 05:37:13 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 05:37:13 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 05:37:13 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 05:37:13 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 05:37:13 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 05:37:13 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 05:37:13 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 05:37:13 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 05:37:13 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 05:37:13 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 05:37:13 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] Mem-Info: [Mon Jul 2 05:37:13 2018] active_anon:591321 inactive_anon:227791 isolated_anon:0 active_file:12233305 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7217152 slab_unreclaimable:3453716 mapped:74342 shmem:228645 pagetables:4195 bounce:0 free:1818679 free_pcp:27 free_cma:0 [Mon Jul 2 05:37:13 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 05:37:13 2018] Node 0 DMA32 free:283368kB min:1184kB low:1480kB high:1776kB active_anon:3260kB inactive_anon:9096kB active_file:50492kB inactive_file:39380kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:700312kB slab_unreclaimable:595564kB kernel_stack:272kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 05:37:13 2018] Node 0 Normal free:247292kB min:43740kB low:54672kB high:65608kB active_anon:337016kB inactive_anon:374140kB active_file:18808868kB inactive_file:18712188kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:143564kB shmem:551072kB slab_reclaimable:18480132kB slab_unreclaimable:5613364kB kernel_stack:4080kB pagetables:4988kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 1 Normal free:6731296kB min:45172kB low:56464kB high:67756kB active_anon:2025008kB inactive_anon:527928kB active_file:30073860kB inactive_file:5101500kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:36kB writeback:0kB mapped:152544kB shmem:361164kB slab_reclaimable:9688164kB slab_unreclaimable:7605872kB kernel_stack:10208kB pagetables:11480kB unstable:0kB bounce:0kB free_pcp:80kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 05:37:13 2018] Node 0 DMA32: 2400*4kB (UEM) 2886*8kB (UEM) 1462*16kB (UEM) 3620*32kB (UEM) 1391*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 283472kB [Mon Jul 2 05:37:13 2018] Node 0 Normal: 37805*4kB (UEM) 5535*8kB (UEM) 2789*16kB (UEM) 304*32kB (UEM) 36*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 252156kB [Mon Jul 2 05:37:13 2018] Node 1 Normal: 234398*4kB (UEM) 240847*8kB (UEM) 241908*16kB (UE) 12*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6735280kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] 18428034 total pagecache pages [Mon Jul 2 05:37:13 2018] 564 pages in swap cache [Mon Jul 2 05:37:13 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 05:37:13 2018] Free swap = 4180704kB [Mon Jul 2 05:37:13 2018] Total swap = 4194300kB [Mon Jul 2 05:37:13 2018] 33530455 pages RAM [Mon Jul 2 05:37:13 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 05:37:13 2018] 594386 pages reserved [Mon Jul 2 05:37:13 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 05:37:13 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 05:37:13 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 05:37:13 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 05:37:13 2018] Call Trace: [Mon Jul 2 05:37:13 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 05:37:13 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 05:37:13 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 05:37:13 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 05:37:13 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 05:37:13 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 05:37:13 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 05:37:13 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 05:37:13 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 05:37:13 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 05:37:13 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 05:37:13 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 05:37:13 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 05:37:13 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 05:37:13 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] Mem-Info: [Mon Jul 2 05:37:13 2018] active_anon:591321 inactive_anon:227791 isolated_anon:0 active_file:12233305 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7217152 slab_unreclaimable:3453716 mapped:74342 shmem:228645 pagetables:4195 bounce:0 free:1818797 free_pcp:0 free_cma:0 [Mon Jul 2 05:37:13 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 05:37:13 2018] Node 0 DMA32 free:283368kB min:1184kB low:1480kB high:1776kB active_anon:3260kB inactive_anon:9096kB active_file:50492kB inactive_file:39380kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:700312kB slab_unreclaimable:595564kB kernel_stack:272kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 05:37:13 2018] Node 0 Normal free:247292kB min:43740kB low:54672kB high:65608kB active_anon:337016kB inactive_anon:374140kB active_file:18808868kB inactive_file:18712188kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:143564kB shmem:551072kB slab_reclaimable:18480132kB slab_unreclaimable:5613364kB kernel_stack:4080kB pagetables:4988kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 1 Normal free:6731768kB min:45172kB low:56464kB high:67756kB active_anon:2025008kB inactive_anon:527928kB active_file:30073860kB inactive_file:5101500kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:36kB writeback:0kB mapped:152544kB shmem:361164kB slab_reclaimable:9688164kB slab_unreclaimable:7605872kB kernel_stack:10208kB pagetables:11480kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 05:37:13 2018] Node 0 DMA32: 2400*4kB (UEM) 2886*8kB (UEM) 1462*16kB (UEM) 3620*32kB (UEM) 1391*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 283472kB [Mon Jul 2 05:37:13 2018] Node 0 Normal: 37806*4kB (UEM) 5537*8kB (UEM) 2789*16kB (UEM) 304*32kB (UEM) 36*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 252176kB [Mon Jul 2 05:37:13 2018] Node 1 Normal: 234407*4kB (UEM) 240848*8kB (UEM) 241921*16kB (UEM) 31*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6736140kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] 18428034 total pagecache pages [Mon Jul 2 05:37:13 2018] 564 pages in swap cache [Mon Jul 2 05:37:13 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 05:37:13 2018] Free swap = 4180704kB [Mon Jul 2 05:37:13 2018] Total swap = 4194300kB [Mon Jul 2 05:37:13 2018] 33530455 pages RAM [Mon Jul 2 05:37:13 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 05:37:13 2018] 594386 pages reserved [Mon Jul 2 05:37:13 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 05:37:13 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 05:37:13 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 05:37:13 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 05:37:13 2018] Call Trace: [Mon Jul 2 05:37:13 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 05:37:13 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 05:37:13 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 05:37:13 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 05:37:13 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 05:37:13 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 05:37:13 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 05:37:13 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 05:37:13 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 05:37:13 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 05:37:13 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 05:37:13 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 05:37:13 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 05:37:13 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 05:37:13 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 05:37:13 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 05:37:13 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 05:37:13 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 05:37:13 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 05:37:13 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 05:37:13 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 05:37:13 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 05:37:13 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 05:37:13 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 05:37:13 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 05:37:13 2018] Mem-Info: [Mon Jul 2 05:37:13 2018] active_anon:591321 inactive_anon:227791 isolated_anon:0 active_file:12233305 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7217152 slab_unreclaimable:3453716 mapped:74342 shmem:228645 pagetables:4195 bounce:0 free:1818797 free_pcp:3 free_cma:0 [Mon Jul 2 05:37:13 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 05:37:13 2018] Node 0 DMA32 free:283368kB min:1184kB low:1480kB high:1776kB active_anon:3260kB inactive_anon:9096kB active_file:50492kB inactive_file:39380kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1260kB shmem:2344kB slab_reclaimable:700312kB slab_unreclaimable:595564kB kernel_stack:272kB pagetables:312kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 05:37:13 2018] Node 0 Normal free:247292kB min:43740kB low:54672kB high:65608kB active_anon:337016kB inactive_anon:374140kB active_file:18808868kB inactive_file:18712188kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:143564kB shmem:551072kB slab_reclaimable:18480132kB slab_unreclaimable:5613364kB kernel_stack:4080kB pagetables:4988kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 1 Normal free:6731768kB min:45172kB low:56464kB high:67756kB active_anon:2025008kB inactive_anon:527928kB active_file:30073860kB inactive_file:5101500kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:36kB writeback:0kB mapped:152544kB shmem:361164kB slab_reclaimable:9688164kB slab_unreclaimable:7605872kB kernel_stack:10208kB pagetables:11480kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 05:37:13 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 05:37:13 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 05:37:13 2018] Node 0 DMA32: 2400*4kB (UEM) 2886*8kB (UEM) 1462*16kB (UEM) 3620*32kB (UEM) 1391*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 283472kB [Mon Jul 2 05:37:13 2018] Node 0 Normal: 37806*4kB (UEM) 5537*8kB (UEM) 2789*16kB (UEM) 304*32kB (UEM) 36*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 252176kB [Mon Jul 2 05:37:13 2018] Node 1 Normal: 234400*4kB (UEM) 240846*8kB (UEM) 241918*16kB (UEM) 30*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6736016kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 05:37:13 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 05:37:13 2018] 18428034 total pagecache pages [Mon Jul 2 05:37:13 2018] 564 pages in swap cache [Mon Jul 2 05:37:13 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 05:37:13 2018] Free swap = 4180704kB [Mon Jul 2 05:37:13 2018] Total swap = 4194300kB [Mon Jul 2 05:37:13 2018] 33530455 pages RAM [Mon Jul 2 05:37:13 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 05:37:13 2018] 594386 pages reserved [Mon Jul 2 05:37:13 2018] LNetError: 65419:0:(o2iblnd.c:934:kiblnd_create_conn()) Can't create QP: -12, send_wr: 409, recv_wr: 4, send_sge: 30, recv_sge: 1 [Mon Jul 2 05:37:13 2018] LNetError: 65419:0:(o2iblnd.c:934:kiblnd_create_conn()) Skipped 1 previous similar message [Mon Jul 2 05:40:12 2018] Lustre: MGS: haven't heard from client 1db370a5-2ae6-d677-ed6e-2bfba780742d (at 172.16.229.20@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882016b45400, cur 1530473808 expire 1530473658 last 1530473581 [Mon Jul 2 05:40:17 2018] Lustre: lustre-MDT0000: haven't heard from client 39fee214-4fae-7d31-5802-27d145d78e9d (at 172.16.229.20@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f43aef000, cur 1530473813 expire 1530473663 last 1530473586 [Mon Jul 2 05:42:35 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 05:42:35 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) Skipped 4288864 previous similar messages [Mon Jul 2 05:42:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 05:42:35 2018] Lustre: Skipped 4288164 previous similar messages [Mon Jul 2 05:42:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 05:42:35 2018] Lustre: Skipped 4288165 previous similar messages [Mon Jul 2 05:52:35 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 05:52:35 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) Skipped 2407908 previous similar messages [Mon Jul 2 05:52:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 05:52:35 2018] Lustre: Skipped 2407935 previous similar messages [Mon Jul 2 05:52:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 05:52:35 2018] Lustre: Skipped 2407935 previous similar messages [Mon Jul 2 06:02:35 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 06:02:35 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) Skipped 2386367 previous similar messages [Mon Jul 2 06:02:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 06:02:35 2018] Lustre: Skipped 2386093 previous similar messages [Mon Jul 2 06:02:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 06:02:35 2018] Lustre: Skipped 2386093 previous similar messages [Mon Jul 2 06:12:35 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 06:12:35 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) Skipped 2405868 previous similar messages [Mon Jul 2 06:12:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 06:12:35 2018] Lustre: Skipped 2406256 previous similar messages [Mon Jul 2 06:12:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 06:12:35 2018] Lustre: Skipped 2406256 previous similar messages [Mon Jul 2 06:23:42 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 06:23:42 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) Skipped 2240788 previous similar messages [Mon Jul 2 06:23:42 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 06:23:42 2018] Lustre: Skipped 2240399 previous similar messages [Mon Jul 2 06:23:42 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 06:23:42 2018] Lustre: Skipped 2240399 previous similar messages [Mon Jul 2 06:33:42 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 06:33:42 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) Skipped 2417604 previous similar messages [Mon Jul 2 06:33:42 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 06:33:42 2018] Lustre: Skipped 2417605 previous similar messages [Mon Jul 2 06:33:42 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 06:33:42 2018] Lustre: Skipped 2417605 previous similar messages [Mon Jul 2 06:43:42 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 06:43:42 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) Skipped 2406522 previous similar messages [Mon Jul 2 06:43:42 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 06:43:42 2018] Lustre: Skipped 2406522 previous similar messages [Mon Jul 2 06:43:42 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 06:43:42 2018] Lustre: Skipped 2406522 previous similar messages [Mon Jul 2 06:53:42 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 06:53:42 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) Skipped 2365344 previous similar messages [Mon Jul 2 06:53:42 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 06:53:42 2018] Lustre: Skipped 2365343 previous similar messages [Mon Jul 2 06:53:42 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 06:53:42 2018] Lustre: Skipped 2365343 previous similar messages [Mon Jul 2 07:03:42 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 07:03:42 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) Skipped 2366340 previous similar messages [Mon Jul 2 07:03:42 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 07:03:42 2018] Lustre: Skipped 2366527 previous similar messages [Mon Jul 2 07:03:42 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 07:03:42 2018] Lustre: Skipped 2366527 previous similar messages [Mon Jul 2 07:13:42 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 07:13:42 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) Skipped 2434011 previous similar messages [Mon Jul 2 07:13:42 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 07:13:42 2018] Lustre: Skipped 2434013 previous similar messages [Mon Jul 2 07:13:42 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 07:13:42 2018] Lustre: Skipped 2434013 previous similar messages [Mon Jul 2 07:23:42 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 07:23:42 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) Skipped 2406602 previous similar messages [Mon Jul 2 07:23:42 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 07:23:42 2018] Lustre: Skipped 2406617 previous similar messages [Mon Jul 2 07:23:42 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 07:23:42 2018] Lustre: Skipped 2406617 previous similar messages [Mon Jul 2 07:35:00 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 07:35:00 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) Skipped 2278676 previous similar messages [Mon Jul 2 07:35:00 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 07:35:00 2018] Lustre: Skipped 2278472 previous similar messages [Mon Jul 2 07:35:00 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 07:35:00 2018] Lustre: Skipped 2278472 previous similar messages [Mon Jul 2 07:45:53 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 07:45:53 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) Skipped 2166566 previous similar messages [Mon Jul 2 07:45:53 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 07:45:53 2018] Lustre: Skipped 2166566 previous similar messages [Mon Jul 2 07:45:53 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 07:45:53 2018] Lustre: Skipped 2166566 previous similar messages [Mon Jul 2 07:55:53 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 07:55:53 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) Skipped 2400390 previous similar messages [Mon Jul 2 07:55:53 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 07:55:53 2018] Lustre: Skipped 2400390 previous similar messages [Mon Jul 2 07:55:53 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 07:55:53 2018] Lustre: Skipped 2400390 previous similar messages [Mon Jul 2 08:05:53 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 08:05:53 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) Skipped 2428637 previous similar messages [Mon Jul 2 08:05:54 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 08:05:54 2018] Lustre: Skipped 2428637 previous similar messages [Mon Jul 2 08:05:54 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 08:05:54 2018] Lustre: Skipped 2428637 previous similar messages [Mon Jul 2 08:15:53 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 08:15:53 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) Skipped 2404545 previous similar messages [Mon Jul 2 08:15:54 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 08:15:54 2018] Lustre: Skipped 2404825 previous similar messages [Mon Jul 2 08:15:54 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 08:15:54 2018] Lustre: Skipped 2404825 previous similar messages [Mon Jul 2 08:25:54 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 08:25:54 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) Skipped 2381052 previous similar messages [Mon Jul 2 08:25:54 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 08:25:54 2018] Lustre: Skipped 2381058 previous similar messages [Mon Jul 2 08:25:54 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 08:25:54 2018] Lustre: Skipped 2381058 previous similar messages [Mon Jul 2 08:35:54 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 08:35:54 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) Skipped 2381290 previous similar messages [Mon Jul 2 08:35:54 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 08:35:54 2018] Lustre: Skipped 2381288 previous similar messages [Mon Jul 2 08:35:54 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 08:35:54 2018] Lustre: Skipped 2381288 previous similar messages [Mon Jul 2 08:41:18 2018] warn_alloc_failed: 6 callbacks suppressed [Mon Jul 2 08:41:18 2018] kworker/11:1: page allocation failure: order:8, mode:0x80d0 [Mon Jul 2 08:41:18 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:41:18 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:41:18 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:41:18 2018] Call Trace: [Mon Jul 2 08:41:18 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:41:18 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:41:18 2018] [] ? on_each_cpu_mask+0x51/0x60 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:41:18 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:41:18 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:41:18 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:41:18 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:41:18 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:41:18 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:41:18 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:41:18 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:41:18 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:41:18 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:41:18 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:41:18 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:41:18 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] Mem-Info: [Mon Jul 2 08:41:18 2018] active_anon:596013 inactive_anon:229838 isolated_anon:0 active_file:12236478 inactive_file:5963269 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7218060 slab_unreclaimable:3455558 mapped:74752 shmem:230693 pagetables:4161 bounce:0 free:1806212 free_pcp:5 free_cma:0 [Mon Jul 2 08:41:18 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:41:18 2018] Node 0 DMA32 free:280552kB min:1184kB low:1480kB high:1776kB active_anon:3792kB inactive_anon:9484kB active_file:50680kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:8kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595644kB kernel_stack:384kB pagetables:244kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:41:18 2018] Node 0 Normal free:231092kB min:43740kB low:54672kB high:65608kB active_anon:341996kB inactive_anon:381564kB active_file:18809240kB inactive_file:18712180kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:144460kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613972kB kernel_stack:3872kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 1 Normal free:6700444kB min:45172kB low:56464kB high:67756kB active_anon:2038264kB inactive_anon:528304kB active_file:30085992kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:28kB writeback:0kB mapped:153324kB shmem:361164kB slab_reclaimable:9687324kB slab_unreclaimable:7612552kB kernel_stack:10288kB pagetables:11640kB unstable:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:41:18 2018] Node 0 DMA32: 2365*4kB (UEM) 2901*8kB (UEM) 1478*16kB (UEM) 3543*32kB (UEM) 1402*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281948kB [Mon Jul 2 08:41:18 2018] Node 0 Normal: 34464*4kB (UEM) 5592*8kB (UM) 2581*16kB (UEM) 301*32kB (UEM) 39*64kB (UM) 2*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 236272kB [Mon Jul 2 08:41:18 2018] Node 1 Normal: 238022*4kB (UEM) 244400*8kB (UEM) 237336*16kB (UEM) 3*32kB (E) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6704760kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] 18433254 total pagecache pages [Mon Jul 2 08:41:18 2018] 564 pages in swap cache [Mon Jul 2 08:41:18 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 08:41:18 2018] Free swap = 4180704kB [Mon Jul 2 08:41:18 2018] Total swap = 4194300kB [Mon Jul 2 08:41:18 2018] 33530455 pages RAM [Mon Jul 2 08:41:18 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:41:18 2018] 594386 pages reserved [Mon Jul 2 08:41:18 2018] kworker/11:1: page allocation failure: order:8, mode:0x80d0 [Mon Jul 2 08:41:18 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:41:18 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:41:18 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:41:18 2018] Call Trace: [Mon Jul 2 08:41:18 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:41:18 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:41:18 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:41:18 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:41:18 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:41:18 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:41:18 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:41:18 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:41:18 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:41:18 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:41:18 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:41:18 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:41:18 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:41:18 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:41:18 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:41:18 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:41:18 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] Mem-Info: [Mon Jul 2 08:41:18 2018] active_anon:596013 inactive_anon:229838 isolated_anon:0 active_file:12236478 inactive_file:5963269 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7218060 slab_unreclaimable:3455558 mapped:74752 shmem:230693 pagetables:4161 bounce:0 free:1806212 free_pcp:0 free_cma:0 [Mon Jul 2 08:41:18 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:41:18 2018] Node 0 DMA32 free:280552kB min:1184kB low:1480kB high:1776kB active_anon:3792kB inactive_anon:9484kB active_file:50680kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:8kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595644kB kernel_stack:384kB pagetables:244kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:41:18 2018] Node 0 Normal free:231092kB min:43740kB low:54672kB high:65608kB active_anon:341996kB inactive_anon:381564kB active_file:18809240kB inactive_file:18712180kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:144460kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613972kB kernel_stack:3872kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 1 Normal free:6700444kB min:45172kB low:56464kB high:67756kB active_anon:2038264kB inactive_anon:528304kB active_file:30085992kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:28kB writeback:0kB mapped:153324kB shmem:361164kB slab_reclaimable:9687324kB slab_unreclaimable:7612552kB kernel_stack:10288kB pagetables:11640kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:41:18 2018] Node 0 DMA32: 2365*4kB (UEM) 2901*8kB (UEM) 1478*16kB (UEM) 3543*32kB (UEM) 1402*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281948kB [Mon Jul 2 08:41:18 2018] Node 0 Normal: 34471*4kB (UEM) 5593*8kB (UEM) 2581*16kB (UEM) 301*32kB (UEM) 39*64kB (UM) 2*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 236308kB [Mon Jul 2 08:41:18 2018] Node 1 Normal: 238017*4kB (UEM) 244398*8kB (UEM) 237335*16kB (UEM) 6*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6704804kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] 18433254 total pagecache pages [Mon Jul 2 08:41:18 2018] 564 pages in swap cache [Mon Jul 2 08:41:18 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 08:41:18 2018] Free swap = 4180704kB [Mon Jul 2 08:41:18 2018] Total swap = 4194300kB [Mon Jul 2 08:41:18 2018] 33530455 pages RAM [Mon Jul 2 08:41:18 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:41:18 2018] 594386 pages reserved [Mon Jul 2 08:41:18 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:41:18 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:41:18 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:41:18 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:41:18 2018] Call Trace: [Mon Jul 2 08:41:18 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:41:18 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:41:18 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:41:18 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:41:18 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:41:18 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:41:18 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:41:18 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:41:18 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:41:18 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:41:18 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:41:18 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:41:18 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:41:18 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:41:18 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] Mem-Info: [Mon Jul 2 08:41:18 2018] active_anon:596081 inactive_anon:229838 isolated_anon:0 active_file:12236478 inactive_file:5963269 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7218060 slab_unreclaimable:3455558 mapped:74752 shmem:230693 pagetables:4161 bounce:0 free:1806182 free_pcp:37 free_cma:0 [Mon Jul 2 08:41:18 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:41:18 2018] Node 0 DMA32 free:280432kB min:1184kB low:1480kB high:1776kB active_anon:4064kB inactive_anon:9484kB active_file:50680kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:8kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595644kB kernel_stack:384kB pagetables:244kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:41:18 2018] Node 0 Normal free:231092kB min:43740kB low:54672kB high:65608kB active_anon:341996kB inactive_anon:381564kB active_file:18809240kB inactive_file:18712180kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:144460kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613972kB kernel_stack:3872kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 1 Normal free:6700444kB min:45172kB low:56464kB high:67756kB active_anon:2038264kB inactive_anon:528304kB active_file:30085992kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:28kB writeback:0kB mapped:153324kB shmem:361164kB slab_reclaimable:9687324kB slab_unreclaimable:7612552kB kernel_stack:10288kB pagetables:11640kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:41:18 2018] Node 0 DMA32: 2292*4kB (UEM) 2901*8kB (UEM) 1478*16kB (UEM) 3543*32kB (UEM) 1402*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281656kB [Mon Jul 2 08:41:18 2018] Node 0 Normal: 34334*4kB (UEM) 5594*8kB (UEM) 2581*16kB (UEM) 300*32kB (UEM) 39*64kB (UM) 2*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 235736kB [Mon Jul 2 08:41:18 2018] Node 1 Normal: 238015*4kB (UEM) 244396*8kB (UEM) 237332*16kB (UEM) 6*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6704732kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] 18433254 total pagecache pages [Mon Jul 2 08:41:18 2018] 564 pages in swap cache [Mon Jul 2 08:41:18 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 08:41:18 2018] Free swap = 4180704kB [Mon Jul 2 08:41:18 2018] Total swap = 4194300kB [Mon Jul 2 08:41:18 2018] 33530455 pages RAM [Mon Jul 2 08:41:18 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:41:18 2018] 594386 pages reserved [Mon Jul 2 08:41:18 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:41:18 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:41:18 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:41:18 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:41:18 2018] Call Trace: [Mon Jul 2 08:41:18 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:41:18 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:41:18 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:41:18 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:41:18 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:41:18 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:41:18 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:41:18 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:41:18 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:41:18 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:41:18 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:41:18 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:41:18 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:41:18 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:41:18 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:41:18 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:41:18 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:41:18 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] Mem-Info: [Mon Jul 2 08:41:18 2018] active_anon:596013 inactive_anon:229838 isolated_anon:0 active_file:12236478 inactive_file:5963269 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7218060 slab_unreclaimable:3455558 mapped:74752 shmem:230693 pagetables:4161 bounce:0 free:1806182 free_pcp:219 free_cma:0 [Mon Jul 2 08:41:18 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:41:18 2018] Node 0 DMA32 free:280432kB min:1184kB low:1480kB high:1776kB active_anon:3792kB inactive_anon:9484kB active_file:50680kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:8kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595644kB kernel_stack:384kB pagetables:244kB unstable:0kB bounce:0kB free_pcp:288kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:41:18 2018] Node 0 Normal free:231092kB min:43740kB low:54672kB high:65608kB active_anon:341996kB inactive_anon:381564kB active_file:18809240kB inactive_file:18712180kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:144460kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613972kB kernel_stack:3872kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:528kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 1 Normal free:6700444kB min:45172kB low:56464kB high:67756kB active_anon:2038264kB inactive_anon:528304kB active_file:30085992kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:28kB writeback:0kB mapped:153324kB shmem:361164kB slab_reclaimable:9687324kB slab_unreclaimable:7612552kB kernel_stack:10288kB pagetables:11640kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:41:18 2018] Node 0 DMA32: 2292*4kB (UEM) 2901*8kB (UEM) 1478*16kB (UEM) 3543*32kB (UEM) 1402*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281656kB [Mon Jul 2 08:41:18 2018] Node 0 Normal: 34320*4kB (UEM) 5594*8kB (UEM) 2581*16kB (UEM) 300*32kB (UEM) 39*64kB (UM) 2*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 235680kB [Mon Jul 2 08:41:18 2018] Node 1 Normal: 238015*4kB (UEM) 244399*8kB (UEM) 237338*16kB (UEM) 6*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6704852kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] 18433254 total pagecache pages [Mon Jul 2 08:41:18 2018] 564 pages in swap cache [Mon Jul 2 08:41:18 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 08:41:18 2018] Free swap = 4180704kB [Mon Jul 2 08:41:18 2018] Total swap = 4194300kB [Mon Jul 2 08:41:18 2018] 33530455 pages RAM [Mon Jul 2 08:41:18 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:41:18 2018] 594386 pages reserved [Mon Jul 2 08:41:18 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:41:18 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:41:18 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:41:18 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:41:18 2018] Call Trace: [Mon Jul 2 08:41:18 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:41:18 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:41:18 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:41:18 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:41:18 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:41:18 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:41:18 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:41:18 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:41:18 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:41:18 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:41:18 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:41:18 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:41:18 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:41:18 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:41:18 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:41:18 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] Mem-Info: [Mon Jul 2 08:41:18 2018] active_anon:596127 inactive_anon:229838 isolated_anon:0 active_file:12236478 inactive_file:5963269 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7218060 slab_unreclaimable:3455558 mapped:74752 shmem:230693 pagetables:4161 bounce:0 free:1806254 free_pcp:0 free_cma:0 [Mon Jul 2 08:41:18 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:41:18 2018] Node 0 DMA32 free:280720kB min:1184kB low:1480kB high:1776kB active_anon:3792kB inactive_anon:9484kB active_file:50680kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:8kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595644kB kernel_stack:384kB pagetables:244kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:41:18 2018] Node 0 Normal free:231092kB min:43740kB low:54672kB high:65608kB active_anon:342452kB inactive_anon:381564kB active_file:18809240kB inactive_file:18712180kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:144460kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613972kB kernel_stack:3872kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:108kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 1 Normal free:6700444kB min:45172kB low:56464kB high:67756kB active_anon:2038264kB inactive_anon:528304kB active_file:30085992kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:28kB writeback:0kB mapped:153324kB shmem:361164kB slab_reclaimable:9687324kB slab_unreclaimable:7612552kB kernel_stack:10288kB pagetables:11640kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:41:18 2018] Node 0 DMA32: 2364*4kB (UEM) 2901*8kB (UEM) 1478*16kB (UEM) 3543*32kB (UEM) 1402*64kB (UEM) 170*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281944kB [Mon Jul 2 08:41:18 2018] Node 0 Normal: 34279*4kB (UEM) 5594*8kB (UEM) 2581*16kB (UEM) 300*32kB (UEM) 39*64kB (UM) 2*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 235516kB [Mon Jul 2 08:41:18 2018] Node 1 Normal: 238017*4kB (UEM) 244398*8kB (UEM) 237340*16kB (UEM) 6*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6704884kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] 18433254 total pagecache pages [Mon Jul 2 08:41:18 2018] 564 pages in swap cache [Mon Jul 2 08:41:18 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 08:41:18 2018] Free swap = 4180704kB [Mon Jul 2 08:41:18 2018] Total swap = 4194300kB [Mon Jul 2 08:41:18 2018] 33530455 pages RAM [Mon Jul 2 08:41:18 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:41:18 2018] 594386 pages reserved [Mon Jul 2 08:41:18 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:41:18 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:41:18 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:41:18 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:41:18 2018] Call Trace: [Mon Jul 2 08:41:18 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:41:18 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:41:18 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:41:18 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:41:18 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:41:18 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:41:18 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:41:18 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:41:18 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:41:18 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:41:18 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:41:18 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:41:18 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:41:18 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:41:18 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:41:18 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:41:18 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] Mem-Info: [Mon Jul 2 08:41:18 2018] active_anon:596113 inactive_anon:229838 isolated_anon:0 active_file:12236478 inactive_file:5963269 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7218060 slab_unreclaimable:3455414 mapped:74752 shmem:230693 pagetables:4162 bounce:0 free:1806959 free_pcp:88 free_cma:0 [Mon Jul 2 08:41:18 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:41:18 2018] Node 0 DMA32 free:280592kB min:1184kB low:1480kB high:1776kB active_anon:3792kB inactive_anon:9484kB active_file:50680kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:8kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595644kB kernel_stack:384kB pagetables:244kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:41:18 2018] Node 0 Normal free:233760kB min:43740kB low:54672kB high:65608kB active_anon:341988kB inactive_anon:381564kB active_file:18809240kB inactive_file:18712180kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:144460kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613396kB kernel_stack:3872kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 1 Normal free:6700724kB min:45172kB low:56464kB high:67756kB active_anon:2038672kB inactive_anon:528304kB active_file:30085992kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:28kB writeback:0kB mapped:153324kB shmem:361164kB slab_reclaimable:9687324kB slab_unreclaimable:7612552kB kernel_stack:10288kB pagetables:11644kB unstable:0kB bounce:0kB free_pcp:352kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:41:18 2018] Node 0 DMA32: 1832*4kB (UEM) 3014*8kB (UEM) 1635*16kB (UEM) 3602*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281920kB [Mon Jul 2 08:41:18 2018] Node 0 Normal: 23541*4kB (UEM) 1780*8kB (UEM) 744*16kB (UEM) 617*32kB (UEM) 136*64kB (UM) 245*128kB (UM) 140*256kB (M) 38*512kB (M) 2*1024kB (M) 0*2048kB 0*4096kB = 237460kB [Mon Jul 2 08:41:18 2018] Node 1 Normal: 238012*4kB (UEM) 244395*8kB (UE) 237256*16kB (UEM) 1*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6703336kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] 18433254 total pagecache pages [Mon Jul 2 08:41:18 2018] 564 pages in swap cache [Mon Jul 2 08:41:18 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 08:41:18 2018] Free swap = 4180704kB [Mon Jul 2 08:41:18 2018] Total swap = 4194300kB [Mon Jul 2 08:41:18 2018] 33530455 pages RAM [Mon Jul 2 08:41:18 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:41:18 2018] 594386 pages reserved [Mon Jul 2 08:41:18 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:41:18 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:41:18 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:41:18 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:41:18 2018] Call Trace: [Mon Jul 2 08:41:18 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:41:18 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:41:18 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:41:18 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:41:18 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:41:18 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:41:18 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:41:18 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:41:18 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:41:18 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:41:18 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:41:18 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:41:18 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:41:18 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:41:18 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] Mem-Info: [Mon Jul 2 08:41:18 2018] active_anon:596299 inactive_anon:229838 isolated_anon:0 active_file:12236478 inactive_file:5963269 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7218060 slab_unreclaimable:3455414 mapped:74752 shmem:230693 pagetables:4162 bounce:0 free:1806752 free_pcp:26 free_cma:0 [Mon Jul 2 08:41:18 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:41:18 2018] Node 0 DMA32 free:280284kB min:1184kB low:1480kB high:1776kB active_anon:4080kB inactive_anon:9484kB active_file:50680kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:8kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595644kB kernel_stack:384kB pagetables:244kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:41:18 2018] Node 0 Normal free:233240kB min:43740kB low:54672kB high:65608kB active_anon:342444kB inactive_anon:381564kB active_file:18809240kB inactive_file:18712180kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:144460kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613396kB kernel_stack:3872kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:224kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 1 Normal free:6700724kB min:45172kB low:56464kB high:67756kB active_anon:2038672kB inactive_anon:528304kB active_file:30085992kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:28kB writeback:0kB mapped:153324kB shmem:361164kB slab_reclaimable:9687324kB slab_unreclaimable:7612552kB kernel_stack:10288kB pagetables:11644kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:41:18 2018] Node 0 DMA32: 1755*4kB (UEM) 3014*8kB (UEM) 1635*16kB (UEM) 3602*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281612kB [Mon Jul 2 08:41:18 2018] Node 0 Normal: 23357*4kB (UEM) 1782*8kB (UEM) 744*16kB (UEM) 616*32kB (UEM) 136*64kB (UM) 245*128kB (UM) 140*256kB (M) 38*512kB (M) 2*1024kB (M) 0*2048kB 0*4096kB = 236708kB [Mon Jul 2 08:41:18 2018] Node 1 Normal: 238016*4kB (UEM) 244406*8kB (UEM) 237269*16kB (UEM) 5*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6703776kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] 18433254 total pagecache pages [Mon Jul 2 08:41:18 2018] 564 pages in swap cache [Mon Jul 2 08:41:18 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 08:41:18 2018] Free swap = 4180704kB [Mon Jul 2 08:41:18 2018] Total swap = 4194300kB [Mon Jul 2 08:41:18 2018] 33530455 pages RAM [Mon Jul 2 08:41:18 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:41:18 2018] 594386 pages reserved [Mon Jul 2 08:41:18 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:41:18 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:41:18 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:41:18 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:41:18 2018] Call Trace: [Mon Jul 2 08:41:18 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:41:18 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:41:18 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:41:18 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:41:18 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:41:18 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:41:18 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:41:18 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:41:18 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:41:18 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:41:18 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:41:18 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:41:18 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:41:18 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:41:18 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:41:18 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:41:18 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:41:18 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] Mem-Info: [Mon Jul 2 08:41:18 2018] active_anon:596113 inactive_anon:229838 isolated_anon:0 active_file:12236478 inactive_file:5963269 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7218060 slab_unreclaimable:3455414 mapped:74752 shmem:230693 pagetables:4162 bounce:0 free:1806942 free_pcp:73 free_cma:0 [Mon Jul 2 08:41:18 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:41:18 2018] Node 0 DMA32 free:280588kB min:1184kB low:1480kB high:1776kB active_anon:3792kB inactive_anon:9484kB active_file:50680kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:8kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595644kB kernel_stack:384kB pagetables:244kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:41:18 2018] Node 0 Normal free:233696kB min:43740kB low:54672kB high:65608kB active_anon:341988kB inactive_anon:381564kB active_file:18809240kB inactive_file:18712180kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:144460kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613396kB kernel_stack:3872kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:260kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 1 Normal free:6700724kB min:45172kB low:56464kB high:67756kB active_anon:2038672kB inactive_anon:528304kB active_file:30085992kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:28kB writeback:0kB mapped:153324kB shmem:361164kB slab_reclaimable:9687324kB slab_unreclaimable:7612552kB kernel_stack:10288kB pagetables:11644kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:41:18 2018] Node 0 DMA32: 1831*4kB (UEM) 3014*8kB (UEM) 1635*16kB (UEM) 3602*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281916kB [Mon Jul 2 08:41:18 2018] Node 0 Normal: 23360*4kB (UEM) 1782*8kB (UEM) 744*16kB (UEM) 616*32kB (UEM) 136*64kB (UM) 245*128kB (UM) 140*256kB (M) 38*512kB (M) 2*1024kB (M) 0*2048kB 0*4096kB = 236720kB [Mon Jul 2 08:41:18 2018] Node 1 Normal: 238016*4kB (UEM) 244406*8kB (UEM) 237269*16kB (UEM) 6*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6703808kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] 18433254 total pagecache pages [Mon Jul 2 08:41:18 2018] 564 pages in swap cache [Mon Jul 2 08:41:18 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 08:41:18 2018] Free swap = 4180704kB [Mon Jul 2 08:41:18 2018] Total swap = 4194300kB [Mon Jul 2 08:41:18 2018] 33530455 pages RAM [Mon Jul 2 08:41:18 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:41:18 2018] 594386 pages reserved [Mon Jul 2 08:41:18 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:41:18 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:41:18 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:41:18 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:41:18 2018] Call Trace: [Mon Jul 2 08:41:18 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:41:18 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:41:18 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:41:18 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:41:18 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:41:18 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:41:18 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:41:18 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:41:18 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:41:18 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:41:18 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:41:18 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:41:18 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:41:18 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:41:18 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:41:18 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] Mem-Info: [Mon Jul 2 08:41:18 2018] active_anon:596227 inactive_anon:229838 isolated_anon:0 active_file:12236476 inactive_file:5963269 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7218060 slab_unreclaimable:3455360 mapped:74752 shmem:230693 pagetables:4162 bounce:0 free:1807000 free_pcp:37 free_cma:0 [Mon Jul 2 08:41:18 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:41:18 2018] Node 0 DMA32 free:280708kB min:1184kB low:1480kB high:1776kB active_anon:3792kB inactive_anon:9484kB active_file:50672kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:8kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595644kB kernel_stack:384kB pagetables:244kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:41:18 2018] Node 0 Normal free:233456kB min:43740kB low:54672kB high:65608kB active_anon:342444kB inactive_anon:381564kB active_file:18809240kB inactive_file:18712180kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:144460kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613348kB kernel_stack:3872kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:300kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 1 Normal free:6701076kB min:45172kB low:56464kB high:67756kB active_anon:2038672kB inactive_anon:528304kB active_file:30085992kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:28kB writeback:0kB mapped:153324kB shmem:361164kB slab_reclaimable:9687324kB slab_unreclaimable:7612384kB kernel_stack:10288kB pagetables:11644kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:41:18 2018] Node 0 DMA32: 1831*4kB (UEM) 3014*8kB (UEM) 1635*16kB (UEM) 3602*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281916kB [Mon Jul 2 08:41:18 2018] Node 0 Normal: 23362*4kB (UEM) 1782*8kB (UEM) 744*16kB (UEM) 616*32kB (UEM) 136*64kB (UM) 245*128kB (UM) 140*256kB (M) 38*512kB (M) 2*1024kB (M) 0*2048kB 0*4096kB = 236728kB [Mon Jul 2 08:41:18 2018] Node 1 Normal: 238017*4kB (UEM) 244406*8kB (UEM) 237269*16kB (UEM) 6*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6703812kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] 18433254 total pagecache pages [Mon Jul 2 08:41:18 2018] 564 pages in swap cache [Mon Jul 2 08:41:18 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 08:41:18 2018] Free swap = 4180704kB [Mon Jul 2 08:41:18 2018] Total swap = 4194300kB [Mon Jul 2 08:41:18 2018] 33530455 pages RAM [Mon Jul 2 08:41:18 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:41:18 2018] 594386 pages reserved [Mon Jul 2 08:41:18 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:41:18 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:41:18 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:41:18 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:41:18 2018] Call Trace: [Mon Jul 2 08:41:18 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:41:18 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:41:18 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:41:18 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:41:18 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:41:18 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:41:18 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:41:18 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:41:18 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:41:18 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:41:18 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:41:18 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:41:18 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:41:18 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:41:18 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:41:18 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:41:18 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:41:18 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:41:18 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:41:18 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:41:18 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:41:18 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:41:18 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:41:18 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:41:18 2018] Mem-Info: [Mon Jul 2 08:41:18 2018] active_anon:596227 inactive_anon:229838 isolated_anon:0 active_file:12236476 inactive_file:5963269 isolated_file:0 unevictable:17363 dirty:11 writeback:0 unstable:0 slab_reclaimable:7218060 slab_unreclaimable:3455360 mapped:74752 shmem:230693 pagetables:4162 bounce:0 free:1807000 free_pcp:37 free_cma:0 [Mon Jul 2 08:41:18 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:41:18 2018] Node 0 DMA32 free:280708kB min:1184kB low:1480kB high:1776kB active_anon:3792kB inactive_anon:9484kB active_file:50672kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:8kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595644kB kernel_stack:384kB pagetables:244kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:41:18 2018] Node 0 Normal free:233456kB min:43740kB low:54672kB high:65608kB active_anon:342444kB inactive_anon:381564kB active_file:18809240kB inactive_file:18712180kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:8kB writeback:0kB mapped:144460kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613348kB kernel_stack:3872kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:684kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 1 Normal free:6701076kB min:45172kB low:56464kB high:67756kB active_anon:2038672kB inactive_anon:528304kB active_file:30085992kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:28kB writeback:0kB mapped:153324kB shmem:361164kB slab_reclaimable:9687324kB slab_unreclaimable:7612384kB kernel_stack:10288kB pagetables:11644kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:41:18 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:41:18 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:41:18 2018] Node 0 DMA32: 1831*4kB (UEM) 3014*8kB (UEM) 1635*16kB (UEM) 3602*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281916kB [Mon Jul 2 08:41:18 2018] Node 0 Normal: 23392*4kB (UEM) 1782*8kB (UEM) 744*16kB (UEM) 616*32kB (UEM) 136*64kB (UM) 245*128kB (UM) 140*256kB (M) 38*512kB (M) 2*1024kB (M) 0*2048kB 0*4096kB = 236848kB [Mon Jul 2 08:41:18 2018] Node 1 Normal: 238022*4kB (UEM) 244405*8kB (UEM) 237267*16kB (UEM) 8*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6703856kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:41:18 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:41:18 2018] 18433254 total pagecache pages [Mon Jul 2 08:41:18 2018] 564 pages in swap cache [Mon Jul 2 08:41:18 2018] Swap cache stats: add 5241, delete 4677, find 853/1003 [Mon Jul 2 08:41:18 2018] Free swap = 4180704kB [Mon Jul 2 08:41:18 2018] Total swap = 4194300kB [Mon Jul 2 08:41:18 2018] 33530455 pages RAM [Mon Jul 2 08:41:18 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:41:18 2018] 594386 pages reserved [Mon Jul 2 08:41:18 2018] LNet: 65419:0:(o2iblnd.c:943:kiblnd_create_conn()) peer 172.16.229.39@o2ib - queue depth reduced from 8 to 1 to allow for qp creation [Mon Jul 2 08:42:10 2018] LNetError: 2851:0:(o2iblnd_cb.c:3251:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds [Mon Jul 2 08:42:10 2018] LNetError: 2851:0:(o2iblnd_cb.c:3326:kiblnd_check_conns()) Timed out RDMA with 172.16.229.39@o2ib (52): c: 0, oc: 1, rc: 1 [Mon Jul 2 08:42:35 2018] warn_alloc_failed: 4 callbacks suppressed [Mon Jul 2 08:42:35 2018] kworker/11:1: page allocation failure: order:8, mode:0x80d0 [Mon Jul 2 08:42:35 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:42:35 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:42:35 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:42:35 2018] Call Trace: [Mon Jul 2 08:42:35 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:42:35 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:42:35 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:42:35 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:42:35 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:42:35 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:42:35 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:42:35 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:42:35 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:42:35 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:42:35 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:42:35 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:42:35 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:42:35 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:42:35 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] Mem-Info: [Mon Jul 2 08:42:35 2018] active_anon:596730 inactive_anon:229838 isolated_anon:0 active_file:12236511 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:61 writeback:0 unstable:0 slab_reclaimable:7217860 slab_unreclaimable:3455428 mapped:74826 shmem:230693 pagetables:4165 bounce:0 free:1806026 free_pcp:5 free_cma:0 [Mon Jul 2 08:42:35 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:42:35 2018] Node 0 DMA32 free:280580kB min:1184kB low:1480kB high:1776kB active_anon:3812kB inactive_anon:9484kB active_file:50696kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595616kB kernel_stack:384kB pagetables:248kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:42:35 2018] Node 0 Normal free:232560kB min:43740kB low:54672kB high:65608kB active_anon:341948kB inactive_anon:381560kB active_file:18809328kB inactive_file:18712172kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:56kB writeback:0kB mapped:144728kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613756kB kernel_stack:3888kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 1 Normal free:6698204kB min:45172kB low:56464kB high:67756kB active_anon:2041160kB inactive_anon:528308kB active_file:30086020kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:188kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686524kB slab_unreclaimable:7612276kB kernel_stack:10288kB pagetables:11652kB unstable:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:42:35 2018] Node 0 DMA32: 1837*4kB (UEM) 3006*8kB (UEM) 1635*16kB (UEM) 3603*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281908kB [Mon Jul 2 08:42:35 2018] Node 0 Normal: 23739*4kB (UEM) 1813*8kB (UEM) 763*16kB (UEM) 559*32kB (UEM) 137*64kB (UM) 247*128kB (UM) 141*256kB (UM) 39*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 237028kB [Mon Jul 2 08:42:35 2018] Node 1 Normal: 238040*4kB (UEM) 244388*8kB (UEM) 237225*16kB (UEM) 2*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6702928kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] 18433286 total pagecache pages [Mon Jul 2 08:42:35 2018] 563 pages in swap cache [Mon Jul 2 08:42:35 2018] Swap cache stats: add 5241, delete 4678, find 853/1003 [Mon Jul 2 08:42:35 2018] Free swap = 4180708kB [Mon Jul 2 08:42:35 2018] Total swap = 4194300kB [Mon Jul 2 08:42:35 2018] 33530455 pages RAM [Mon Jul 2 08:42:35 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:42:35 2018] 594386 pages reserved [Mon Jul 2 08:42:35 2018] kworker/11:1: page allocation failure: order:8, mode:0x80d0 [Mon Jul 2 08:42:35 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:42:35 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:42:35 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:42:35 2018] Call Trace: [Mon Jul 2 08:42:35 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:42:35 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:42:35 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:42:35 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:42:35 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:42:35 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:42:35 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:42:35 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:42:35 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:42:35 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:42:35 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:42:35 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:42:35 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:42:35 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:42:35 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:42:35 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:42:35 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:42:35 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] Mem-Info: [Mon Jul 2 08:42:35 2018] active_anon:596730 inactive_anon:229838 isolated_anon:0 active_file:12236511 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:61 writeback:0 unstable:0 slab_reclaimable:7217860 slab_unreclaimable:3455428 mapped:74826 shmem:230693 pagetables:4165 bounce:0 free:1806026 free_pcp:6 free_cma:0 [Mon Jul 2 08:42:35 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:42:35 2018] Node 0 DMA32 free:280580kB min:1184kB low:1480kB high:1776kB active_anon:3812kB inactive_anon:9484kB active_file:50696kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595616kB kernel_stack:384kB pagetables:248kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:42:35 2018] Node 0 Normal free:232560kB min:43740kB low:54672kB high:65608kB active_anon:341948kB inactive_anon:381560kB active_file:18809328kB inactive_file:18712172kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:56kB writeback:0kB mapped:144728kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613756kB kernel_stack:3888kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 1 Normal free:6698204kB min:45172kB low:56464kB high:67756kB active_anon:2041160kB inactive_anon:528308kB active_file:30086020kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:188kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686524kB slab_unreclaimable:7612276kB kernel_stack:10288kB pagetables:11652kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:42:35 2018] Node 0 DMA32: 1837*4kB (UEM) 3006*8kB (UEM) 1635*16kB (UEM) 3603*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281908kB [Mon Jul 2 08:42:35 2018] Node 0 Normal: 23741*4kB (UEM) 1813*8kB (UEM) 763*16kB (UEM) 559*32kB (UEM) 137*64kB (UM) 247*128kB (UM) 141*256kB (UM) 39*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 237036kB [Mon Jul 2 08:42:35 2018] Node 1 Normal: 238038*4kB (UEM) 244387*8kB (UEM) 237227*16kB (UEM) 3*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6702976kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] 18433286 total pagecache pages [Mon Jul 2 08:42:35 2018] 563 pages in swap cache [Mon Jul 2 08:42:35 2018] Swap cache stats: add 5241, delete 4678, find 853/1003 [Mon Jul 2 08:42:35 2018] Free swap = 4180708kB [Mon Jul 2 08:42:35 2018] Total swap = 4194300kB [Mon Jul 2 08:42:35 2018] 33530455 pages RAM [Mon Jul 2 08:42:35 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:42:35 2018] 594386 pages reserved [Mon Jul 2 08:42:35 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:42:35 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:42:35 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:42:35 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:42:35 2018] Call Trace: [Mon Jul 2 08:42:35 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:42:35 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:42:35 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:42:35 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:42:35 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:42:35 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:42:35 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:42:35 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:42:35 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:42:35 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:42:35 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:42:35 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:42:35 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:42:35 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:42:35 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:42:35 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] Mem-Info: [Mon Jul 2 08:42:35 2018] active_anon:596730 inactive_anon:229838 isolated_anon:0 active_file:12236511 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:61 writeback:0 unstable:0 slab_reclaimable:7217860 slab_unreclaimable:3455428 mapped:74826 shmem:230693 pagetables:4165 bounce:0 free:1806026 free_pcp:2 free_cma:0 [Mon Jul 2 08:42:35 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:42:35 2018] Node 0 DMA32 free:280580kB min:1184kB low:1480kB high:1776kB active_anon:3812kB inactive_anon:9484kB active_file:50696kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595616kB kernel_stack:384kB pagetables:248kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:42:35 2018] Node 0 Normal free:232560kB min:43740kB low:54672kB high:65608kB active_anon:341948kB inactive_anon:381560kB active_file:18809328kB inactive_file:18712172kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:56kB writeback:0kB mapped:144728kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613756kB kernel_stack:3888kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 1 Normal free:6698204kB min:45172kB low:56464kB high:67756kB active_anon:2041160kB inactive_anon:528308kB active_file:30086020kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:188kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686524kB slab_unreclaimable:7612276kB kernel_stack:10288kB pagetables:11652kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:42:35 2018] Node 0 DMA32: 1837*4kB (UEM) 3006*8kB (UEM) 1635*16kB (UEM) 3603*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281908kB [Mon Jul 2 08:42:35 2018] Node 0 Normal: 23744*4kB (UEM) 1813*8kB (UEM) 763*16kB (UEM) 559*32kB (UEM) 137*64kB (UM) 247*128kB (UM) 141*256kB (UM) 39*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 237048kB [Mon Jul 2 08:42:35 2018] Node 1 Normal: 238040*4kB (UEM) 244387*8kB (UEM) 237228*16kB (UEM) 3*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6703000kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] 18433286 total pagecache pages [Mon Jul 2 08:42:35 2018] 563 pages in swap cache [Mon Jul 2 08:42:35 2018] Swap cache stats: add 5241, delete 4678, find 853/1003 [Mon Jul 2 08:42:35 2018] Free swap = 4180708kB [Mon Jul 2 08:42:35 2018] Total swap = 4194300kB [Mon Jul 2 08:42:35 2018] 33530455 pages RAM [Mon Jul 2 08:42:35 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:42:35 2018] 594386 pages reserved [Mon Jul 2 08:42:35 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:42:35 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:42:35 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:42:35 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:42:35 2018] Call Trace: [Mon Jul 2 08:42:35 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:42:35 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:42:35 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:42:35 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:42:35 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:42:35 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:42:35 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:42:35 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:42:35 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:42:35 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:42:35 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:42:35 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:42:35 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:42:35 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:42:35 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:42:35 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:42:35 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:42:35 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] Mem-Info: [Mon Jul 2 08:42:35 2018] active_anon:596730 inactive_anon:229838 isolated_anon:0 active_file:12236511 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:61 writeback:0 unstable:0 slab_reclaimable:7217860 slab_unreclaimable:3455428 mapped:74826 shmem:230693 pagetables:4165 bounce:0 free:1806026 free_pcp:4 free_cma:0 [Mon Jul 2 08:42:35 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:42:35 2018] Node 0 DMA32 free:280580kB min:1184kB low:1480kB high:1776kB active_anon:3812kB inactive_anon:9484kB active_file:50696kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595616kB kernel_stack:384kB pagetables:248kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:42:35 2018] Node 0 Normal free:232560kB min:43740kB low:54672kB high:65608kB active_anon:341948kB inactive_anon:381560kB active_file:18809328kB inactive_file:18712172kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:56kB writeback:0kB mapped:144728kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613756kB kernel_stack:3888kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 1 Normal free:6698204kB min:45172kB low:56464kB high:67756kB active_anon:2041160kB inactive_anon:528308kB active_file:30086020kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:188kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686524kB slab_unreclaimable:7612276kB kernel_stack:10288kB pagetables:11652kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:42:35 2018] Node 0 DMA32: 1837*4kB (UEM) 3006*8kB (UEM) 1635*16kB (UEM) 3603*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281908kB [Mon Jul 2 08:42:35 2018] Node 0 Normal: 23747*4kB (UEM) 1813*8kB (UEM) 763*16kB (UEM) 559*32kB (UEM) 137*64kB (UM) 247*128kB (UM) 141*256kB (UM) 39*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 237060kB [Mon Jul 2 08:42:35 2018] Node 1 Normal: 238042*4kB (UEM) 244387*8kB (UEM) 237232*16kB (UEM) 3*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6703072kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] 18433286 total pagecache pages [Mon Jul 2 08:42:35 2018] 563 pages in swap cache [Mon Jul 2 08:42:35 2018] Swap cache stats: add 5241, delete 4678, find 853/1003 [Mon Jul 2 08:42:35 2018] Free swap = 4180708kB [Mon Jul 2 08:42:35 2018] Total swap = 4194300kB [Mon Jul 2 08:42:35 2018] 33530455 pages RAM [Mon Jul 2 08:42:35 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:42:35 2018] 594386 pages reserved [Mon Jul 2 08:42:35 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:42:35 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:42:35 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:42:35 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:42:35 2018] Call Trace: [Mon Jul 2 08:42:35 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:42:35 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:42:35 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:42:35 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:42:35 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:42:35 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:42:35 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:42:35 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:42:35 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:42:35 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:42:35 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:42:35 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:42:35 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:42:35 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:42:35 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:42:35 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] Mem-Info: [Mon Jul 2 08:42:35 2018] active_anon:596730 inactive_anon:229838 isolated_anon:0 active_file:12236511 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:61 writeback:0 unstable:0 slab_reclaimable:7217860 slab_unreclaimable:3455428 mapped:74826 shmem:230693 pagetables:4165 bounce:0 free:1805985 free_pcp:59 free_cma:0 [Mon Jul 2 08:42:35 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:42:35 2018] Node 0 DMA32 free:280416kB min:1184kB low:1480kB high:1776kB active_anon:3812kB inactive_anon:9484kB active_file:50696kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595616kB kernel_stack:384kB pagetables:248kB unstable:0kB bounce:0kB free_pcp:212kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:42:35 2018] Node 0 Normal free:232560kB min:43740kB low:54672kB high:65608kB active_anon:341948kB inactive_anon:381560kB active_file:18809328kB inactive_file:18712172kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:56kB writeback:0kB mapped:144728kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613756kB kernel_stack:3888kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 1 Normal free:6698204kB min:45172kB low:56464kB high:67756kB active_anon:2041160kB inactive_anon:528308kB active_file:30086020kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:188kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686524kB slab_unreclaimable:7612276kB kernel_stack:10288kB pagetables:11652kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:42:35 2018] Node 0 DMA32: 1774*4kB (UEM) 3006*8kB (UEM) 1635*16kB (UEM) 3603*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281656kB [Mon Jul 2 08:42:35 2018] Node 0 Normal: 23748*4kB (UEM) 1813*8kB (UEM) 763*16kB (UEM) 559*32kB (UEM) 137*64kB (UM) 247*128kB (UM) 141*256kB (UM) 39*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 237064kB [Mon Jul 2 08:42:35 2018] Node 1 Normal: 238044*4kB (UEM) 244386*8kB (UEM) 237233*16kB (UEM) 4*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6703120kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] 18433286 total pagecache pages [Mon Jul 2 08:42:35 2018] 563 pages in swap cache [Mon Jul 2 08:42:35 2018] Swap cache stats: add 5241, delete 4678, find 853/1003 [Mon Jul 2 08:42:35 2018] Free swap = 4180708kB [Mon Jul 2 08:42:35 2018] Total swap = 4194300kB [Mon Jul 2 08:42:35 2018] 33530455 pages RAM [Mon Jul 2 08:42:35 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:42:35 2018] 594386 pages reserved [Mon Jul 2 08:42:35 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:42:35 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:42:35 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:42:35 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:42:35 2018] Call Trace: [Mon Jul 2 08:42:35 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:42:35 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:42:35 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:42:35 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:42:35 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:42:35 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:42:35 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:42:35 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:42:35 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:42:35 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:42:35 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:42:35 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:42:35 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:42:35 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:42:35 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:42:35 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:42:35 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] Mem-Info: [Mon Jul 2 08:42:35 2018] active_anon:596904 inactive_anon:229838 isolated_anon:0 active_file:12236511 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:61 writeback:0 unstable:0 slab_reclaimable:7217860 slab_unreclaimable:3455428 mapped:74826 shmem:230693 pagetables:4165 bounce:0 free:1805862 free_pcp:43 free_cma:0 [Mon Jul 2 08:42:35 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:42:35 2018] Node 0 DMA32 free:280404kB min:1184kB low:1480kB high:1776kB active_anon:3892kB inactive_anon:9484kB active_file:50696kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595616kB kernel_stack:384kB pagetables:248kB unstable:0kB bounce:0kB free_pcp:160kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:42:35 2018] Node 0 Normal free:232080kB min:43740kB low:54672kB high:65608kB active_anon:342404kB inactive_anon:381560kB active_file:18809328kB inactive_file:18712172kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:56kB writeback:0kB mapped:144728kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613756kB kernel_stack:3888kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:408kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 1 Normal free:6698204kB min:45172kB low:56464kB high:67756kB active_anon:2041160kB inactive_anon:528308kB active_file:30086020kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:188kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686524kB slab_unreclaimable:7612276kB kernel_stack:10288kB pagetables:11652kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:42:35 2018] Node 0 DMA32: 1772*4kB (UEM) 3006*8kB (UEM) 1635*16kB (UEM) 3603*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281648kB [Mon Jul 2 08:42:35 2018] Node 0 Normal: 23567*4kB (UEM) 1813*8kB (UEM) 763*16kB (UEM) 559*32kB (UEM) 137*64kB (UM) 247*128kB (UM) 141*256kB (UM) 39*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 236340kB [Mon Jul 2 08:42:35 2018] Node 1 Normal: 238045*4kB (UEM) 244385*8kB (UEM) 237233*16kB (UEM) 5*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6703148kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] 18433286 total pagecache pages [Mon Jul 2 08:42:35 2018] 563 pages in swap cache [Mon Jul 2 08:42:35 2018] Swap cache stats: add 5241, delete 4678, find 853/1003 [Mon Jul 2 08:42:35 2018] Free swap = 4180708kB [Mon Jul 2 08:42:35 2018] Total swap = 4194300kB [Mon Jul 2 08:42:35 2018] 33530455 pages RAM [Mon Jul 2 08:42:35 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:42:35 2018] 594386 pages reserved [Mon Jul 2 08:42:35 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:42:35 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:42:35 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:42:35 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:42:35 2018] Call Trace: [Mon Jul 2 08:42:35 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:42:35 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:42:35 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:42:35 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:42:35 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:42:35 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:42:35 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:42:35 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:42:35 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:42:35 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:42:35 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:42:35 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:42:35 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:42:35 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:42:35 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:42:35 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] Mem-Info: [Mon Jul 2 08:42:35 2018] active_anon:596730 inactive_anon:229838 isolated_anon:0 active_file:12236511 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:61 writeback:0 unstable:0 slab_reclaimable:7217860 slab_unreclaimable:3455428 mapped:74826 shmem:230693 pagetables:4165 bounce:0 free:1805926 free_pcp:27 free_cma:0 [Mon Jul 2 08:42:35 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:42:35 2018] Node 0 DMA32 free:280660kB min:1184kB low:1480kB high:1776kB active_anon:3812kB inactive_anon:9484kB active_file:50696kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595616kB kernel_stack:384kB pagetables:248kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:42:35 2018] Node 0 Normal free:232080kB min:43740kB low:54672kB high:65608kB active_anon:341948kB inactive_anon:381560kB active_file:18809328kB inactive_file:18712172kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:56kB writeback:0kB mapped:144728kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613756kB kernel_stack:3888kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:80kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 1 Normal free:6698204kB min:45172kB low:56464kB high:67756kB active_anon:2041160kB inactive_anon:528308kB active_file:30086020kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:188kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686524kB slab_unreclaimable:7612276kB kernel_stack:10288kB pagetables:11652kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:42:35 2018] Node 0 DMA32: 1838*4kB (UEM) 3006*8kB (UEM) 1635*16kB (UEM) 3603*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281912kB [Mon Jul 2 08:42:35 2018] Node 0 Normal: 23618*4kB (UEM) 1813*8kB (UEM) 763*16kB (UEM) 559*32kB (UEM) 137*64kB (UM) 247*128kB (UM) 141*256kB (UM) 39*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 236544kB [Mon Jul 2 08:42:35 2018] Node 1 Normal: 238045*4kB (UEM) 244385*8kB (UEM) 237233*16kB (UEM) 6*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6703180kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] 18433286 total pagecache pages [Mon Jul 2 08:42:35 2018] 563 pages in swap cache [Mon Jul 2 08:42:35 2018] Swap cache stats: add 5241, delete 4678, find 853/1003 [Mon Jul 2 08:42:35 2018] Free swap = 4180708kB [Mon Jul 2 08:42:35 2018] Total swap = 4194300kB [Mon Jul 2 08:42:35 2018] 33530455 pages RAM [Mon Jul 2 08:42:35 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:42:35 2018] 594386 pages reserved [Mon Jul 2 08:42:35 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:42:35 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:42:35 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:42:35 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:42:35 2018] Call Trace: [Mon Jul 2 08:42:35 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:42:35 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:42:35 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:42:35 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:42:35 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:42:35 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:42:35 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:42:35 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:42:35 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:42:35 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:42:35 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:42:35 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:42:35 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:42:35 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:42:35 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:42:35 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:42:35 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:42:35 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] Mem-Info: [Mon Jul 2 08:42:35 2018] active_anon:596730 inactive_anon:229838 isolated_anon:0 active_file:12236511 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:61 writeback:0 unstable:0 slab_reclaimable:7217860 slab_unreclaimable:3455428 mapped:74826 shmem:230693 pagetables:4165 bounce:0 free:1805809 free_pcp:178 free_cma:0 [Mon Jul 2 08:42:35 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:42:35 2018] Node 0 DMA32 free:280660kB min:1184kB low:1480kB high:1776kB active_anon:3812kB inactive_anon:9484kB active_file:50696kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595616kB kernel_stack:384kB pagetables:248kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:42:35 2018] Node 0 Normal free:231612kB min:43740kB low:54672kB high:65608kB active_anon:341948kB inactive_anon:381560kB active_file:18809328kB inactive_file:18712172kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:56kB writeback:0kB mapped:144728kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613756kB kernel_stack:3888kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:712kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 1 Normal free:6698204kB min:45172kB low:56464kB high:67756kB active_anon:2041160kB inactive_anon:528308kB active_file:30086020kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:188kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686524kB slab_unreclaimable:7612276kB kernel_stack:10288kB pagetables:11652kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:42:35 2018] Node 0 DMA32: 1838*4kB (UEM) 3006*8kB (UEM) 1635*16kB (UEM) 3603*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281912kB [Mon Jul 2 08:42:35 2018] Node 0 Normal: 23578*4kB (UEM) 1813*8kB (UEM) 763*16kB (UEM) 559*32kB (UEM) 137*64kB (UM) 247*128kB (UM) 141*256kB (UM) 39*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 236384kB [Mon Jul 2 08:42:35 2018] Node 1 Normal: 238045*4kB (UEM) 244384*8kB (UEM) 237231*16kB (UEM) 8*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6703204kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] 18433286 total pagecache pages [Mon Jul 2 08:42:35 2018] 563 pages in swap cache [Mon Jul 2 08:42:35 2018] Swap cache stats: add 5241, delete 4678, find 853/1003 [Mon Jul 2 08:42:35 2018] Free swap = 4180708kB [Mon Jul 2 08:42:35 2018] Total swap = 4194300kB [Mon Jul 2 08:42:35 2018] 33530455 pages RAM [Mon Jul 2 08:42:35 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:42:35 2018] 594386 pages reserved [Mon Jul 2 08:42:35 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:42:35 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:42:35 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:42:35 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:42:35 2018] Call Trace: [Mon Jul 2 08:42:35 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:42:35 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:42:35 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:42:35 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:42:35 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:42:35 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:42:35 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:42:35 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:42:35 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:42:35 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:42:35 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:42:35 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:42:35 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:42:35 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:42:35 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] Mem-Info: [Mon Jul 2 08:42:35 2018] active_anon:596730 inactive_anon:229838 isolated_anon:0 active_file:12236511 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:61 writeback:0 unstable:0 slab_reclaimable:7217860 slab_unreclaimable:3455428 mapped:74826 shmem:230693 pagetables:4165 bounce:0 free:1805809 free_pcp:163 free_cma:0 [Mon Jul 2 08:42:35 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:42:35 2018] Node 0 DMA32 free:280660kB min:1184kB low:1480kB high:1776kB active_anon:3812kB inactive_anon:9484kB active_file:50696kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595616kB kernel_stack:384kB pagetables:248kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:42:35 2018] Node 0 Normal free:231612kB min:43740kB low:54672kB high:65608kB active_anon:341948kB inactive_anon:381560kB active_file:18809328kB inactive_file:18712172kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:56kB writeback:0kB mapped:144728kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613756kB kernel_stack:3888kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:652kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 1 Normal free:6698204kB min:45172kB low:56464kB high:67756kB active_anon:2041160kB inactive_anon:528308kB active_file:30086020kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:188kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686524kB slab_unreclaimable:7612276kB kernel_stack:10288kB pagetables:11652kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:42:35 2018] Node 0 DMA32: 1838*4kB (UEM) 3006*8kB (UEM) 1635*16kB (UEM) 3603*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281912kB [Mon Jul 2 08:42:35 2018] Node 0 Normal: 23594*4kB (UEM) 1813*8kB (UEM) 763*16kB (UEM) 559*32kB (UEM) 137*64kB (UM) 247*128kB (UM) 141*256kB (UM) 39*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 236448kB [Mon Jul 2 08:42:35 2018] Node 1 Normal: 238047*4kB (UEM) 244384*8kB (UEM) 237232*16kB (UEM) 8*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6703228kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] 18433286 total pagecache pages [Mon Jul 2 08:42:35 2018] 563 pages in swap cache [Mon Jul 2 08:42:35 2018] Swap cache stats: add 5241, delete 4678, find 853/1003 [Mon Jul 2 08:42:35 2018] Free swap = 4180708kB [Mon Jul 2 08:42:35 2018] Total swap = 4194300kB [Mon Jul 2 08:42:35 2018] 33530455 pages RAM [Mon Jul 2 08:42:35 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:42:35 2018] 594386 pages reserved [Mon Jul 2 08:42:35 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:42:35 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:42:35 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:42:35 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:42:35 2018] Call Trace: [Mon Jul 2 08:42:35 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:42:35 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:42:35 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:42:35 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:42:35 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:42:35 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:42:35 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:42:35 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:42:35 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:42:35 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:42:35 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:42:35 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:42:35 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:42:35 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:42:35 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:42:35 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:42:35 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:42:35 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:42:35 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:42:35 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:42:35 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:42:35 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:42:35 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:42:35 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:42:35 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:42:35 2018] Mem-Info: [Mon Jul 2 08:42:35 2018] active_anon:596730 inactive_anon:229838 isolated_anon:0 active_file:12236511 inactive_file:5963267 isolated_file:0 unevictable:17363 dirty:61 writeback:0 unstable:0 slab_reclaimable:7217860 slab_unreclaimable:3455428 mapped:74826 shmem:230693 pagetables:4165 bounce:0 free:1806156 free_pcp:33 free_cma:0 [Mon Jul 2 08:42:35 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:42:35 2018] Node 0 DMA32 free:280660kB min:1184kB low:1480kB high:1776kB active_anon:3812kB inactive_anon:9484kB active_file:50696kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595616kB kernel_stack:384kB pagetables:248kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:42:35 2018] Node 0 Normal free:232524kB min:43740kB low:54672kB high:65608kB active_anon:341948kB inactive_anon:381560kB active_file:18809328kB inactive_file:18712172kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:56kB writeback:0kB mapped:144728kB shmem:558856kB slab_reclaimable:18484452kB slab_unreclaimable:5613756kB kernel_stack:3888kB pagetables:4760kB unstable:0kB bounce:0kB free_pcp:128kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 1 Normal free:6698680kB min:45172kB low:56464kB high:67756kB active_anon:2041160kB inactive_anon:528308kB active_file:30086020kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:188kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686524kB slab_unreclaimable:7612276kB kernel_stack:10288kB pagetables:11652kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:42:35 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:42:35 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:42:35 2018] Node 0 DMA32: 1838*4kB (UEM) 3006*8kB (UEM) 1635*16kB (UEM) 3603*32kB (UEM) 1368*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281912kB [Mon Jul 2 08:42:35 2018] Node 0 Normal: 23726*4kB (UEM) 1813*8kB (UEM) 763*16kB (UEM) 559*32kB (UEM) 137*64kB (UM) 247*128kB (UM) 141*256kB (UM) 39*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 236976kB [Mon Jul 2 08:42:35 2018] Node 1 Normal: 238047*4kB (UEM) 244384*8kB (UEM) 237232*16kB (UEM) 10*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6703292kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:42:35 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:42:35 2018] 18433286 total pagecache pages [Mon Jul 2 08:42:35 2018] 563 pages in swap cache [Mon Jul 2 08:42:35 2018] Swap cache stats: add 5241, delete 4678, find 853/1003 [Mon Jul 2 08:42:35 2018] Free swap = 4180708kB [Mon Jul 2 08:42:35 2018] Total swap = 4194300kB [Mon Jul 2 08:42:35 2018] 33530455 pages RAM [Mon Jul 2 08:42:35 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:42:35 2018] 594386 pages reserved [Mon Jul 2 08:42:35 2018] LNet: 65419:0:(o2iblnd.c:943:kiblnd_create_conn()) peer 172.16.229.39@o2ib - queue depth reduced from 8 to 1 to allow for qp creation [Mon Jul 2 08:42:35 2018] Lustre: MGS: Received new LWP connection from 172.16.229.39@o2ib, removing former export from same NID [Mon Jul 2 08:43:26 2018] LNetError: 2851:0:(o2iblnd_cb.c:3251:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [Mon Jul 2 08:43:26 2018] LNetError: 2851:0:(o2iblnd_cb.c:3326:kiblnd_check_conns()) Timed out RDMA with 172.16.229.39@o2ib (51): c: 0, oc: 1, rc: 1 [Mon Jul 2 08:43:51 2018] warn_alloc_failed: 4 callbacks suppressed [Mon Jul 2 08:43:51 2018] kworker/11:1: page allocation failure: order:8, mode:0x80d0 [Mon Jul 2 08:43:51 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:43:51 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:43:51 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:43:51 2018] Call Trace: [Mon Jul 2 08:43:51 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:43:51 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:43:51 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:43:51 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:43:51 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:43:51 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:43:51 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:43:51 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:43:51 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:43:51 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:43:51 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:43:51 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:43:51 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:43:51 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:43:51 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] Mem-Info: [Mon Jul 2 08:43:51 2018] active_anon:597671 inactive_anon:229838 isolated_anon:0 active_file:12236544 inactive_file:5963268 isolated_file:0 unevictable:17363 dirty:4 writeback:0 unstable:0 slab_reclaimable:7217846 slab_unreclaimable:3455059 mapped:74889 shmem:230693 pagetables:4351 bounce:0 free:1805490 free_pcp:0 free_cma:0 [Mon Jul 2 08:43:51 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:43:51 2018] Node 0 DMA32 free:281844kB min:1184kB low:1480kB high:1776kB active_anon:3952kB inactive_anon:9484kB active_file:50700kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595608kB kernel_stack:400kB pagetables:260kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:43:51 2018] Node 0 Normal free:231372kB min:43740kB low:54672kB high:65608kB active_anon:342420kB inactive_anon:381560kB active_file:18809424kB inactive_file:18712176kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:4kB writeback:0kB mapped:144980kB shmem:558856kB slab_reclaimable:18484460kB slab_unreclaimable:5612828kB kernel_stack:3872kB pagetables:5356kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 1 Normal free:6695984kB min:45172kB low:56464kB high:67756kB active_anon:2044312kB inactive_anon:528308kB active_file:30086052kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:12kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686460kB slab_unreclaimable:7611736kB kernel_stack:10368kB pagetables:11788kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:43:51 2018] Node 0 DMA32: 1861*4kB (UEM) 3011*8kB (UEM) 1644*16kB (UEM) 3603*32kB (UEM) 1364*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281932kB [Mon Jul 2 08:43:51 2018] Node 0 Normal: 23614*4kB (UEM) 1802*8kB (UEM) 745*16kB (UEM) 563*32kB (UEM) 138*64kB (UM) 248*128kB (UM) 143*256kB (UM) 38*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 236472kB [Mon Jul 2 08:43:51 2018] Node 1 Normal: 238070*4kB (UEM) 244450*8kB (UEM) 237057*16kB (UEM) 4*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6700920kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] 18433318 total pagecache pages [Mon Jul 2 08:43:51 2018] 562 pages in swap cache [Mon Jul 2 08:43:51 2018] Swap cache stats: add 5241, delete 4679, find 853/1003 [Mon Jul 2 08:43:51 2018] Free swap = 4180712kB [Mon Jul 2 08:43:51 2018] Total swap = 4194300kB [Mon Jul 2 08:43:51 2018] 33530455 pages RAM [Mon Jul 2 08:43:51 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:43:51 2018] 594386 pages reserved [Mon Jul 2 08:43:51 2018] kworker/11:1: page allocation failure: order:8, mode:0x80d0 [Mon Jul 2 08:43:51 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:43:51 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:43:51 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:43:51 2018] Call Trace: [Mon Jul 2 08:43:51 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:43:51 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:43:51 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:43:51 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:43:51 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:43:51 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:43:51 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:43:51 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:43:51 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:43:51 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:43:51 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:43:51 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:43:51 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:43:51 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:43:51 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:43:51 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:43:51 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:43:51 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] Mem-Info: [Mon Jul 2 08:43:51 2018] active_anon:597671 inactive_anon:229838 isolated_anon:0 active_file:12236544 inactive_file:5963268 isolated_file:0 unevictable:17363 dirty:4 writeback:0 unstable:0 slab_reclaimable:7217846 slab_unreclaimable:3455059 mapped:74889 shmem:230693 pagetables:4351 bounce:0 free:1805490 free_pcp:4 free_cma:0 [Mon Jul 2 08:43:51 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:43:51 2018] Node 0 DMA32 free:281844kB min:1184kB low:1480kB high:1776kB active_anon:3952kB inactive_anon:9484kB active_file:50700kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595608kB kernel_stack:400kB pagetables:260kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:43:51 2018] Node 0 Normal free:231372kB min:43740kB low:54672kB high:65608kB active_anon:342420kB inactive_anon:381560kB active_file:18809424kB inactive_file:18712176kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:4kB writeback:0kB mapped:144980kB shmem:558856kB slab_reclaimable:18484460kB slab_unreclaimable:5612828kB kernel_stack:3872kB pagetables:5356kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 1 Normal free:6695984kB min:45172kB low:56464kB high:67756kB active_anon:2044312kB inactive_anon:528308kB active_file:30086052kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:12kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686460kB slab_unreclaimable:7611736kB kernel_stack:10368kB pagetables:11788kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:43:51 2018] Node 0 DMA32: 1861*4kB (UEM) 3011*8kB (UEM) 1644*16kB (UEM) 3603*32kB (UEM) 1364*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281932kB [Mon Jul 2 08:43:51 2018] Node 0 Normal: 23614*4kB (UEM) 1804*8kB (UEM) 745*16kB (UEM) 563*32kB (UEM) 138*64kB (UM) 248*128kB (UM) 143*256kB (UM) 38*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 236488kB [Mon Jul 2 08:43:51 2018] Node 1 Normal: 238070*4kB (UEM) 244448*8kB (UEM) 237062*16kB (UEM) 4*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6700984kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] 18433318 total pagecache pages [Mon Jul 2 08:43:51 2018] 562 pages in swap cache [Mon Jul 2 08:43:51 2018] Swap cache stats: add 5241, delete 4679, find 853/1003 [Mon Jul 2 08:43:51 2018] Free swap = 4180712kB [Mon Jul 2 08:43:51 2018] Total swap = 4194300kB [Mon Jul 2 08:43:51 2018] 33530455 pages RAM [Mon Jul 2 08:43:51 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:43:51 2018] 594386 pages reserved [Mon Jul 2 08:43:51 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:43:51 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:43:51 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:43:51 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:43:51 2018] Call Trace: [Mon Jul 2 08:43:51 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:43:51 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:43:51 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:43:51 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:43:51 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:43:51 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:43:51 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:43:51 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:43:51 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:43:51 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:43:51 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:43:51 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:43:51 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:43:51 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:43:51 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:43:51 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] Mem-Info: [Mon Jul 2 08:43:51 2018] active_anon:597671 inactive_anon:229838 isolated_anon:0 active_file:12236544 inactive_file:5963268 isolated_file:0 unevictable:17363 dirty:4 writeback:0 unstable:0 slab_reclaimable:7217846 slab_unreclaimable:3455059 mapped:74889 shmem:230693 pagetables:4351 bounce:0 free:1805490 free_pcp:6 free_cma:0 [Mon Jul 2 08:43:51 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:43:51 2018] Node 0 DMA32 free:281844kB min:1184kB low:1480kB high:1776kB active_anon:3952kB inactive_anon:9484kB active_file:50700kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595608kB kernel_stack:400kB pagetables:260kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:43:51 2018] Node 0 Normal free:231372kB min:43740kB low:54672kB high:65608kB active_anon:342420kB inactive_anon:381560kB active_file:18809424kB inactive_file:18712176kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:4kB writeback:0kB mapped:144980kB shmem:558856kB slab_reclaimable:18484460kB slab_unreclaimable:5612828kB kernel_stack:3872kB pagetables:5356kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 1 Normal free:6695984kB min:45172kB low:56464kB high:67756kB active_anon:2044312kB inactive_anon:528308kB active_file:30086052kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:12kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686460kB slab_unreclaimable:7611736kB kernel_stack:10368kB pagetables:11788kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:43:51 2018] Node 0 DMA32: 1861*4kB (UEM) 3011*8kB (UEM) 1644*16kB (UEM) 3603*32kB (UEM) 1364*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281932kB [Mon Jul 2 08:43:51 2018] Node 0 Normal: 23617*4kB (UEM) 1804*8kB (UEM) 745*16kB (UEM) 563*32kB (UEM) 138*64kB (UM) 248*128kB (UM) 143*256kB (UM) 38*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 236500kB [Mon Jul 2 08:43:51 2018] Node 1 Normal: 238072*4kB (UEM) 244449*8kB (UEM) 237065*16kB (UEM) 4*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6701048kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] 18433318 total pagecache pages [Mon Jul 2 08:43:51 2018] 562 pages in swap cache [Mon Jul 2 08:43:51 2018] Swap cache stats: add 5241, delete 4679, find 853/1003 [Mon Jul 2 08:43:51 2018] Free swap = 4180712kB [Mon Jul 2 08:43:51 2018] Total swap = 4194300kB [Mon Jul 2 08:43:51 2018] 33530455 pages RAM [Mon Jul 2 08:43:51 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:43:51 2018] 594386 pages reserved [Mon Jul 2 08:43:51 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:43:51 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:43:51 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:43:51 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:43:51 2018] Call Trace: [Mon Jul 2 08:43:51 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:43:51 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:43:51 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:43:51 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:43:51 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:43:51 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:43:51 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:43:51 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:43:51 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:43:51 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:43:51 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:43:51 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:43:51 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:43:51 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:43:51 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:43:51 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:43:51 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:43:51 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] Mem-Info: [Mon Jul 2 08:43:51 2018] active_anon:597714 inactive_anon:229838 isolated_anon:0 active_file:12236544 inactive_file:5963268 isolated_file:0 unevictable:17363 dirty:4 writeback:0 unstable:0 slab_reclaimable:7217846 slab_unreclaimable:3455059 mapped:74889 shmem:230693 pagetables:4351 bounce:0 free:1805323 free_pcp:61 free_cma:0 [Mon Jul 2 08:43:51 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:43:51 2018] Node 0 DMA32 free:281724kB min:1184kB low:1480kB high:1776kB active_anon:4124kB inactive_anon:9484kB active_file:50700kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595608kB kernel_stack:400kB pagetables:260kB unstable:0kB bounce:0kB free_pcp:60kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:43:51 2018] Node 0 Normal free:230824kB min:43740kB low:54672kB high:65608kB active_anon:342420kB inactive_anon:381560kB active_file:18809424kB inactive_file:18712176kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:4kB writeback:0kB mapped:144980kB shmem:558856kB slab_reclaimable:18484460kB slab_unreclaimable:5612828kB kernel_stack:3872kB pagetables:5356kB unstable:0kB bounce:0kB free_pcp:172kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 1 Normal free:6695984kB min:45172kB low:56464kB high:67756kB active_anon:2044312kB inactive_anon:528308kB active_file:30086052kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:12kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686460kB slab_unreclaimable:7611736kB kernel_stack:10368kB pagetables:11788kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:43:51 2018] Node 0 DMA32: 1830*4kB (UEM) 3011*8kB (UEM) 1644*16kB (UEM) 3603*32kB (UEM) 1364*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281808kB [Mon Jul 2 08:43:51 2018] Node 0 Normal: 23514*4kB (UEM) 1804*8kB (UEM) 745*16kB (UEM) 563*32kB (UEM) 138*64kB (UM) 248*128kB (UM) 143*256kB (UM) 38*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 236088kB [Mon Jul 2 08:43:51 2018] Node 1 Normal: 238076*4kB (UEM) 244449*8kB (UEM) 237065*16kB (UEM) 4*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6701064kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] 18433318 total pagecache pages [Mon Jul 2 08:43:51 2018] 562 pages in swap cache [Mon Jul 2 08:43:51 2018] Swap cache stats: add 5241, delete 4679, find 853/1003 [Mon Jul 2 08:43:51 2018] Free swap = 4180712kB [Mon Jul 2 08:43:51 2018] Total swap = 4194300kB [Mon Jul 2 08:43:51 2018] 33530455 pages RAM [Mon Jul 2 08:43:51 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:43:51 2018] 594386 pages reserved [Mon Jul 2 08:43:51 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:43:51 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:43:51 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:43:51 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:43:51 2018] Call Trace: [Mon Jul 2 08:43:51 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:43:51 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:43:51 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:43:51 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:43:51 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:43:51 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:43:51 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:43:51 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:43:51 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:43:51 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:43:51 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:43:51 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:43:51 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:43:51 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:43:51 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] Mem-Info: [Mon Jul 2 08:43:51 2018] active_anon:597820 inactive_anon:229838 isolated_anon:0 active_file:12236544 inactive_file:5963268 isolated_file:0 unevictable:17363 dirty:4 writeback:0 unstable:0 slab_reclaimable:7217846 slab_unreclaimable:3455059 mapped:74889 shmem:230693 pagetables:4351 bounce:0 free:1805353 free_pcp:49 free_cma:0 [Mon Jul 2 08:43:51 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:43:51 2018] Node 0 DMA32 free:281844kB min:1184kB low:1480kB high:1776kB active_anon:4092kB inactive_anon:9484kB active_file:50700kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595608kB kernel_stack:400kB pagetables:260kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:43:51 2018] Node 0 Normal free:230824kB min:43740kB low:54672kB high:65608kB active_anon:342876kB inactive_anon:381560kB active_file:18809424kB inactive_file:18712176kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:4kB writeback:0kB mapped:144980kB shmem:558856kB slab_reclaimable:18484460kB slab_unreclaimable:5612828kB kernel_stack:3872kB pagetables:5356kB unstable:0kB bounce:0kB free_pcp:168kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 1 Normal free:6695984kB min:45172kB low:56464kB high:67756kB active_anon:2044312kB inactive_anon:528308kB active_file:30086052kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:12kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686460kB slab_unreclaimable:7611736kB kernel_stack:10368kB pagetables:11788kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:43:51 2018] Node 0 DMA32: 1861*4kB (UEM) 3011*8kB (UEM) 1644*16kB (UEM) 3603*32kB (UEM) 1364*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281932kB [Mon Jul 2 08:43:51 2018] Node 0 Normal: 23497*4kB (UEM) 1804*8kB (UEM) 745*16kB (UEM) 563*32kB (UEM) 138*64kB (UM) 248*128kB (UM) 143*256kB (UM) 38*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 236020kB [Mon Jul 2 08:43:51 2018] Node 1 Normal: 238079*4kB (UEM) 244450*8kB (UEM) 237068*16kB (UEM) 4*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6701132kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] 18433318 total pagecache pages [Mon Jul 2 08:43:51 2018] 562 pages in swap cache [Mon Jul 2 08:43:51 2018] Swap cache stats: add 5241, delete 4679, find 853/1003 [Mon Jul 2 08:43:51 2018] Free swap = 4180712kB [Mon Jul 2 08:43:51 2018] Total swap = 4194300kB [Mon Jul 2 08:43:51 2018] 33530455 pages RAM [Mon Jul 2 08:43:51 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:43:51 2018] 594386 pages reserved [Mon Jul 2 08:43:51 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:43:51 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:43:51 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:43:51 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:43:51 2018] Call Trace: [Mon Jul 2 08:43:51 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:43:51 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:43:51 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:43:51 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:43:51 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:43:51 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:43:51 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:43:51 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:43:51 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:43:51 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:43:51 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:43:51 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:43:51 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:43:51 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:43:51 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:43:51 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:43:51 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:43:51 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] Mem-Info: [Mon Jul 2 08:43:51 2018] active_anon:597820 inactive_anon:229838 isolated_anon:0 active_file:12236544 inactive_file:5963268 isolated_file:0 unevictable:17363 dirty:4 writeback:0 unstable:0 slab_reclaimable:7217846 slab_unreclaimable:3455059 mapped:74889 shmem:230693 pagetables:4351 bounce:0 free:1805353 free_pcp:6 free_cma:0 [Mon Jul 2 08:43:51 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:43:51 2018] Node 0 DMA32 free:281844kB min:1184kB low:1480kB high:1776kB active_anon:4092kB inactive_anon:9484kB active_file:50700kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595608kB kernel_stack:400kB pagetables:260kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:43:51 2018] Node 0 Normal free:230824kB min:43740kB low:54672kB high:65608kB active_anon:342876kB inactive_anon:381560kB active_file:18809424kB inactive_file:18712176kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:4kB writeback:0kB mapped:144980kB shmem:558856kB slab_reclaimable:18484460kB slab_unreclaimable:5612828kB kernel_stack:3872kB pagetables:5356kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 1 Normal free:6695984kB min:45172kB low:56464kB high:67756kB active_anon:2044312kB inactive_anon:528308kB active_file:30086052kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:12kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686460kB slab_unreclaimable:7611736kB kernel_stack:10368kB pagetables:11788kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:43:51 2018] Node 0 DMA32: 1861*4kB (UEM) 3011*8kB (UEM) 1644*16kB (UEM) 3603*32kB (UEM) 1364*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281932kB [Mon Jul 2 08:43:51 2018] Node 0 Normal: 23422*4kB (UEM) 1804*8kB (UEM) 745*16kB (UEM) 563*32kB (UEM) 138*64kB (UM) 248*128kB (UM) 143*256kB (UM) 38*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 235720kB [Mon Jul 2 08:43:51 2018] Node 1 Normal: 238079*4kB (UEM) 244450*8kB (UEM) 237068*16kB (UEM) 4*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6701132kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] 18433318 total pagecache pages [Mon Jul 2 08:43:51 2018] 562 pages in swap cache [Mon Jul 2 08:43:51 2018] Swap cache stats: add 5241, delete 4679, find 853/1003 [Mon Jul 2 08:43:51 2018] Free swap = 4180712kB [Mon Jul 2 08:43:51 2018] Total swap = 4194300kB [Mon Jul 2 08:43:51 2018] 33530455 pages RAM [Mon Jul 2 08:43:51 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:43:51 2018] 594386 pages reserved [Mon Jul 2 08:43:51 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:43:51 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:43:51 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:43:51 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:43:51 2018] Call Trace: [Mon Jul 2 08:43:51 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:43:51 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:43:51 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:43:51 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:43:51 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:43:51 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:43:51 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:43:51 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:43:51 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:43:51 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:43:51 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:43:51 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:43:51 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:43:51 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:43:51 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:43:51 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] Mem-Info: [Mon Jul 2 08:43:51 2018] active_anon:597820 inactive_anon:229838 isolated_anon:0 active_file:12236544 inactive_file:5963268 isolated_file:0 unevictable:17363 dirty:4 writeback:0 unstable:0 slab_reclaimable:7217846 slab_unreclaimable:3455059 mapped:74889 shmem:230693 pagetables:4351 bounce:0 free:1805230 free_pcp:122 free_cma:0 [Mon Jul 2 08:43:51 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:43:51 2018] Node 0 DMA32 free:281844kB min:1184kB low:1480kB high:1776kB active_anon:4092kB inactive_anon:9484kB active_file:50700kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595608kB kernel_stack:400kB pagetables:260kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:43:51 2018] Node 0 Normal free:230332kB min:43740kB low:54672kB high:65608kB active_anon:342876kB inactive_anon:381560kB active_file:18809424kB inactive_file:18712176kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:4kB writeback:0kB mapped:144980kB shmem:558856kB slab_reclaimable:18484460kB slab_unreclaimable:5612828kB kernel_stack:3872kB pagetables:5356kB unstable:0kB bounce:0kB free_pcp:460kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 1 Normal free:6695984kB min:45172kB low:56464kB high:67756kB active_anon:2044312kB inactive_anon:528308kB active_file:30086052kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:12kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686460kB slab_unreclaimable:7611736kB kernel_stack:10368kB pagetables:11788kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:43:51 2018] Node 0 DMA32: 1861*4kB (UEM) 3011*8kB (UEM) 1644*16kB (UEM) 3603*32kB (UEM) 1364*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281932kB [Mon Jul 2 08:43:51 2018] Node 0 Normal: 23443*4kB (UEM) 1804*8kB (UEM) 745*16kB (UEM) 563*32kB (UEM) 138*64kB (UM) 248*128kB (UM) 143*256kB (UM) 38*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 235804kB [Mon Jul 2 08:43:51 2018] Node 1 Normal: 238080*4kB (UEM) 244451*8kB (UEM) 237068*16kB (UEM) 4*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6701144kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] 18433318 total pagecache pages [Mon Jul 2 08:43:51 2018] 562 pages in swap cache [Mon Jul 2 08:43:51 2018] Swap cache stats: add 5241, delete 4679, find 853/1003 [Mon Jul 2 08:43:51 2018] Free swap = 4180712kB [Mon Jul 2 08:43:51 2018] Total swap = 4194300kB [Mon Jul 2 08:43:51 2018] 33530455 pages RAM [Mon Jul 2 08:43:51 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:43:51 2018] 594386 pages reserved [Mon Jul 2 08:43:51 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:43:51 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:43:51 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:43:51 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:43:51 2018] Call Trace: [Mon Jul 2 08:43:51 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:43:51 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:43:51 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:43:51 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:43:51 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:43:51 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:43:51 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:43:51 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:43:51 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:43:51 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:43:51 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:43:51 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:43:51 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:43:51 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:43:51 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:43:51 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:43:51 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] Mem-Info: [Mon Jul 2 08:43:51 2018] active_anon:597820 inactive_anon:229838 isolated_anon:0 active_file:12236544 inactive_file:5963268 isolated_file:0 unevictable:17363 dirty:4 writeback:0 unstable:0 slab_reclaimable:7217846 slab_unreclaimable:3455059 mapped:74889 shmem:230693 pagetables:4351 bounce:0 free:1805230 free_pcp:156 free_cma:0 [Mon Jul 2 08:43:51 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:43:51 2018] Node 0 DMA32 free:281844kB min:1184kB low:1480kB high:1776kB active_anon:4092kB inactive_anon:9484kB active_file:50700kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595608kB kernel_stack:400kB pagetables:260kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:43:51 2018] Node 0 Normal free:230332kB min:43740kB low:54672kB high:65608kB active_anon:342876kB inactive_anon:381560kB active_file:18809424kB inactive_file:18712176kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:4kB writeback:0kB mapped:144980kB shmem:558856kB slab_reclaimable:18484460kB slab_unreclaimable:5612828kB kernel_stack:3872kB pagetables:5356kB unstable:0kB bounce:0kB free_pcp:624kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 1 Normal free:6695984kB min:45172kB low:56464kB high:67756kB active_anon:2044312kB inactive_anon:528308kB active_file:30086052kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:12kB writeback:0kB mapped:153352kB shmem:361164kB slab_reclaimable:9686460kB slab_unreclaimable:7611736kB kernel_stack:10368kB pagetables:11788kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:43:51 2018] Node 0 DMA32: 1861*4kB (UEM) 3011*8kB (UEM) 1644*16kB (UEM) 3603*32kB (UEM) 1364*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281932kB [Mon Jul 2 08:43:51 2018] Node 0 Normal: 23469*4kB (UEM) 1805*8kB (UEM) 745*16kB (UEM) 563*32kB (UEM) 138*64kB (UM) 248*128kB (UM) 143*256kB (UM) 38*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 235916kB [Mon Jul 2 08:43:51 2018] Node 1 Normal: 238083*4kB (UEM) 244450*8kB (UEM) 237071*16kB (UEM) 5*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6701228kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] 18433318 total pagecache pages [Mon Jul 2 08:43:51 2018] 562 pages in swap cache [Mon Jul 2 08:43:51 2018] Swap cache stats: add 5241, delete 4679, find 853/1003 [Mon Jul 2 08:43:51 2018] Free swap = 4180712kB [Mon Jul 2 08:43:51 2018] Total swap = 4194300kB [Mon Jul 2 08:43:51 2018] 33530455 pages RAM [Mon Jul 2 08:43:51 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:43:51 2018] 594386 pages reserved [Mon Jul 2 08:43:51 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:43:51 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:43:51 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:43:51 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:43:51 2018] Call Trace: [Mon Jul 2 08:43:51 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:43:51 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:43:51 2018] [] ? drain_pages+0xb0/0xb0 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:43:51 2018] [] dma_generic_alloc_coherent+0x8f/0x140 [Mon Jul 2 08:43:51 2018] [] x86_swiotlb_alloc_coherent+0x21/0x50 [Mon Jul 2 08:43:51 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:43:51 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:43:51 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:43:51 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:43:51 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:43:51 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:43:51 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:43:51 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:43:51 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:43:51 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:43:51 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] Mem-Info: [Mon Jul 2 08:43:51 2018] active_anon:597632 inactive_anon:229837 isolated_anon:0 active_file:12236544 inactive_file:5963268 isolated_file:0 unevictable:17363 dirty:8 writeback:0 unstable:0 slab_reclaimable:7217846 slab_unreclaimable:3455123 mapped:74897 shmem:230692 pagetables:4281 bounce:0 free:1807461 free_pcp:7 free_cma:0 [Mon Jul 2 08:43:51 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:43:51 2018] Node 0 DMA32 free:281908kB min:1184kB low:1480kB high:1776kB active_anon:3884kB inactive_anon:9484kB active_file:50700kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595608kB kernel_stack:384kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:43:51 2018] Node 0 Normal free:235608kB min:43740kB low:54672kB high:65608kB active_anon:342508kB inactive_anon:381556kB active_file:18809424kB inactive_file:18712176kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:16kB writeback:0kB mapped:144980kB shmem:558852kB slab_reclaimable:18484460kB slab_unreclaimable:5613084kB kernel_stack:3872kB pagetables:5180kB unstable:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 1 Normal free:6699568kB min:45172kB low:56464kB high:67756kB active_anon:2044136kB inactive_anon:528308kB active_file:30086052kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:16kB writeback:0kB mapped:153384kB shmem:361164kB slab_reclaimable:9686460kB slab_unreclaimable:7611736kB kernel_stack:10336kB pagetables:11704kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:43:51 2018] Node 0 DMA32: 1861*4kB (UEM) 3011*8kB (UEM) 1644*16kB (UEM) 3603*32kB (UEM) 1364*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281932kB [Mon Jul 2 08:43:51 2018] Node 0 Normal: 23625*4kB (UEM) 1806*8kB (UEM) 745*16kB (UEM) 563*32kB (UEM) 138*64kB (UM) 248*128kB (UM) 143*256kB (UM) 38*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 236548kB [Mon Jul 2 08:43:51 2018] Node 1 Normal: 238083*4kB (UEM) 244449*8kB (UEM) 237071*16kB (UEM) 6*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6701252kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] 18433319 total pagecache pages [Mon Jul 2 08:43:51 2018] 562 pages in swap cache [Mon Jul 2 08:43:51 2018] Swap cache stats: add 5241, delete 4679, find 853/1003 [Mon Jul 2 08:43:51 2018] Free swap = 4180712kB [Mon Jul 2 08:43:51 2018] Total swap = 4194300kB [Mon Jul 2 08:43:51 2018] 33530455 pages RAM [Mon Jul 2 08:43:51 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:43:51 2018] 594386 pages reserved [Mon Jul 2 08:43:51 2018] kworker/11:1: page allocation failure: order:9, mode:0x80d0 [Mon Jul 2 08:43:51 2018] CPU: 11 PID: 65419 Comm: kworker/11:1 Tainted: G OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1 [Mon Jul 2 08:43:51 2018] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.3.4 11/08/2016 [Mon Jul 2 08:43:51 2018] Workqueue: ib_cm cm_work_handler [ib_cm] [Mon Jul 2 08:43:51 2018] Call Trace: [Mon Jul 2 08:43:51 2018] [] dump_stack+0x19/0x1b [Mon Jul 2 08:43:51 2018] [] warn_alloc_failed+0x110/0x180 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_slowpath+0x6b6/0x724 [Mon Jul 2 08:43:51 2018] [] __alloc_pages_nodemask+0x405/0x420 [Mon Jul 2 08:43:51 2018] [] alloc_pages_current+0x98/0x110 [Mon Jul 2 08:43:51 2018] [] __get_free_pages+0xe/0x40 [Mon Jul 2 08:43:51 2018] [] swiotlb_alloc_coherent+0x5e/0x150 [Mon Jul 2 08:43:51 2018] [] x86_swiotlb_alloc_coherent+0x41/0x50 [Mon Jul 2 08:43:51 2018] [] mlx5_dma_zalloc_coherent_node+0xad/0x110 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc_node+0x3e/0xa0 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] mlx5_buf_alloc+0x14/0x20 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] create_kernel_qp.isra.65+0x44d/0x76d [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] create_qp_common+0x9e8/0x1660 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? debugfs_create_file+0x1f/0x30 [Mon Jul 2 08:43:51 2018] [] ? mlx5_debug_cq_add+0x4b/0x70 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? mlx5_core_create_cq+0x1ae/0x230 [mlx5_core] [Mon Jul 2 08:43:51 2018] [] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 2 08:43:51 2018] [] ? _mlx5_ib_create_qp+0xfd/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] _mlx5_ib_create_qp+0x126/0x530 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ? backport_kvfree+0x35/0x40 [mlx_compat] [Mon Jul 2 08:43:51 2018] [] ? mlx5_ib_create_cq+0x300/0x4c0 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] mlx5_ib_create_qp+0x10/0x20 [mlx5_ib] [Mon Jul 2 08:43:51 2018] [] ib_create_qp+0x7a/0x2f0 [ib_core] [Mon Jul 2 08:43:51 2018] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_create_conn+0xbff/0x1870 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Mon Jul 2 08:43:51 2018] [] kiblnd_passive_connect+0xa4f/0x1790 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] kiblnd_cm_callback+0x755/0x2390 [ko2iblnd] [Mon Jul 2 08:43:51 2018] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [Mon Jul 2 08:43:51 2018] [] cm_process_work+0x27/0x120 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [Mon Jul 2 08:43:51 2018] [] cm_work_handler+0x395/0x1306 [ib_cm] [Mon Jul 2 08:43:51 2018] [] ? __schedule+0x424/0x9b0 [Mon Jul 2 08:43:51 2018] [] process_one_work+0x17a/0x440 [Mon Jul 2 08:43:51 2018] [] worker_thread+0x126/0x3c0 [Mon Jul 2 08:43:51 2018] [] ? manage_workers.isra.24+0x2a0/0x2a0 [Mon Jul 2 08:43:51 2018] [] kthread+0xcf/0xe0 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] [] ret_from_fork+0x58/0x90 [Mon Jul 2 08:43:51 2018] [] ? insert_kthread_work+0x40/0x40 [Mon Jul 2 08:43:51 2018] Mem-Info: [Mon Jul 2 08:43:51 2018] active_anon:597632 inactive_anon:229837 isolated_anon:0 active_file:12236544 inactive_file:5963268 isolated_file:0 unevictable:17363 dirty:8 writeback:0 unstable:0 slab_reclaimable:7217846 slab_unreclaimable:3454981 mapped:74897 shmem:230692 pagetables:4281 bounce:0 free:1807711 free_pcp:22 free_cma:0 [Mon Jul 2 08:43:51 2018] Node 0 DMA free:12760kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:64kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 1690 64141 64141 [Mon Jul 2 08:43:51 2018] Node 0 DMA32 free:281916kB min:1184kB low:1480kB high:1776kB active_anon:3884kB inactive_anon:9484kB active_file:50700kB inactive_file:39384kB unevictable:3464kB isolated(anon):0kB isolated(file):0kB present:1985264kB managed:1733076kB mlocked:3464kB dirty:0kB writeback:0kB mapped:1224kB shmem:2752kB slab_reclaimable:700464kB slab_unreclaimable:595608kB kernel_stack:384kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 62450 62450 [Mon Jul 2 08:43:51 2018] Node 0 Normal free:235676kB min:43740kB low:54672kB high:65608kB active_anon:342508kB inactive_anon:381556kB active_file:18809424kB inactive_file:18712176kB unevictable:50456kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63949200kB mlocked:50456kB dirty:16kB writeback:0kB mapped:144980kB shmem:558852kB slab_reclaimable:18484460kB slab_unreclaimable:5612976kB kernel_stack:3872kB pagetables:5180kB unstable:0kB bounce:0kB free_pcp:64kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 1 Normal free:6700492kB min:45172kB low:56464kB high:67756kB active_anon:2044136kB inactive_anon:528308kB active_file:30086052kB inactive_file:5101512kB unevictable:15532kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66046104kB mlocked:15532kB dirty:16kB writeback:0kB mapped:153384kB shmem:361164kB slab_reclaimable:9686460kB slab_unreclaimable:7611276kB kernel_stack:10336kB pagetables:11704kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [Mon Jul 2 08:43:51 2018] lowmem_reserve[]: 0 0 0 0 [Mon Jul 2 08:43:51 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 12760kB [Mon Jul 2 08:43:51 2018] Node 0 DMA32: 1861*4kB (UEM) 3011*8kB (UEM) 1644*16kB (UEM) 3603*32kB (UEM) 1364*64kB (UEM) 162*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 281932kB [Mon Jul 2 08:43:51 2018] Node 0 Normal: 23456*4kB (UEM) 1806*8kB (UEM) 745*16kB (UEM) 562*32kB (UEM) 138*64kB (UM) 248*128kB (UM) 143*256kB (UM) 38*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 235840kB [Mon Jul 2 08:43:51 2018] Node 1 Normal: 238086*4kB (UEM) 244448*8kB (UEM) 237071*16kB (UEM) 7*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6701288kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Mon Jul 2 08:43:51 2018] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Mon Jul 2 08:43:51 2018] 18433319 total pagecache pages [Mon Jul 2 08:43:51 2018] 562 pages in swap cache [Mon Jul 2 08:43:51 2018] Swap cache stats: add 5241, delete 4679, find 853/1003 [Mon Jul 2 08:43:51 2018] Free swap = 4180712kB [Mon Jul 2 08:43:51 2018] Total swap = 4194300kB [Mon Jul 2 08:43:51 2018] 33530455 pages RAM [Mon Jul 2 08:43:51 2018] 0 pages HighMem/MovableOnly [Mon Jul 2 08:43:51 2018] 594386 pages reserved [Mon Jul 2 08:43:51 2018] LNet: 65419:0:(o2iblnd.c:943:kiblnd_create_conn()) peer 172.16.229.39@o2ib - queue depth reduced from 8 to 1 to allow for qp creation [Mon Jul 2 08:43:51 2018] Lustre: MGS: Received new LWP connection from 172.16.229.39@o2ib, removing former export from same NID [Mon Jul 2 08:44:54 2018] LNetError: 2851:0:(o2iblnd_cb.c:3251:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 4 seconds [Mon Jul 2 08:44:54 2018] LNetError: 2851:0:(o2iblnd_cb.c:3326:kiblnd_check_conns()) Timed out RDMA with 172.16.229.39@o2ib (55): c: 0, oc: 1, rc: 1 [Mon Jul 2 08:45:06 2018] bash (133875): drop_caches: 1 [Mon Jul 2 08:45:21 2018] Lustre: MGS: Received new LWP connection from 172.16.229.39@o2ib, removing former export from same NID [Mon Jul 2 08:45:54 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 08:45:54 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) Skipped 2375987 previous similar messages [Mon Jul 2 08:45:54 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 08:45:54 2018] Lustre: Skipped 2375896 previous similar messages [Mon Jul 2 08:45:54 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 08:45:54 2018] Lustre: Skipped 2375901 previous similar messages [Mon Jul 2 08:55:54 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 08:55:54 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) Skipped 2503505 previous similar messages [Mon Jul 2 08:55:54 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 08:55:54 2018] Lustre: Skipped 2503506 previous similar messages [Mon Jul 2 08:55:54 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 08:55:54 2018] Lustre: Skipped 2503506 previous similar messages [Mon Jul 2 09:05:54 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 09:05:54 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) Skipped 2437694 previous similar messages [Mon Jul 2 09:05:54 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 09:05:54 2018] Lustre: Skipped 2437666 previous similar messages [Mon Jul 2 09:05:54 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 09:05:54 2018] Lustre: Skipped 2437666 previous similar messages [Mon Jul 2 09:15:54 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 09:15:54 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) Skipped 2413317 previous similar messages [Mon Jul 2 09:15:54 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 09:15:54 2018] Lustre: Skipped 2413357 previous similar messages [Mon Jul 2 09:15:54 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 09:15:54 2018] Lustre: Skipped 2413357 previous similar messages [Mon Jul 2 09:25:54 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 09:25:54 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) Skipped 2440305 previous similar messages [Mon Jul 2 09:25:54 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 09:25:54 2018] Lustre: Skipped 2440386 previous similar messages [Mon Jul 2 09:25:54 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 09:25:54 2018] Lustre: Skipped 2440386 previous similar messages [Mon Jul 2 09:35:54 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 09:35:54 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) Skipped 1935846 previous similar messages [Mon Jul 2 09:35:54 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 09:35:54 2018] Lustre: Skipped 1935766 previous similar messages [Mon Jul 2 09:35:54 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 09:35:54 2018] Lustre: Skipped 1935766 previous similar messages [Mon Jul 2 09:45:54 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 09:45:54 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) Skipped 2421026 previous similar messages [Mon Jul 2 09:45:54 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 09:45:54 2018] Lustre: Skipped 2421026 previous similar messages [Mon Jul 2 09:45:54 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 09:45:54 2018] Lustre: Skipped 2421026 previous similar messages [Mon Jul 2 09:55:54 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 09:55:54 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) Skipped 2410323 previous similar messages [Mon Jul 2 09:55:54 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 09:55:54 2018] Lustre: Skipped 2410117 previous similar messages [Mon Jul 2 09:55:54 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 09:55:54 2018] Lustre: Skipped 2410117 previous similar messages [Mon Jul 2 10:05:54 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 10:05:54 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) Skipped 1415631 previous similar messages [Mon Jul 2 10:05:54 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 10:05:54 2018] Lustre: Skipped 1416143 previous similar messages [Mon Jul 2 10:05:54 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 10:05:54 2018] Lustre: Skipped 1416143 previous similar messages [Mon Jul 2 10:16:09 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 10:16:09 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 1512725 previous similar messages [Mon Jul 2 10:16:09 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 10:16:09 2018] Lustre: Skipped 1512212 previous similar messages [Mon Jul 2 10:16:09 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 10:16:09 2018] Lustre: Skipped 1512212 previous similar messages [Mon Jul 2 10:26:09 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 10:26:09 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) Skipped 2352499 previous similar messages [Mon Jul 2 10:26:09 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 10:26:09 2018] Lustre: Skipped 2352499 previous similar messages [Mon Jul 2 10:26:09 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 10:26:09 2018] Lustre: Skipped 2352499 previous similar messages [Mon Jul 2 10:36:09 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 10:36:09 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) Skipped 2382779 previous similar messages [Mon Jul 2 10:36:09 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 10:36:09 2018] Lustre: Skipped 2382821 previous similar messages [Mon Jul 2 10:36:09 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 10:36:09 2018] Lustre: Skipped 2382821 previous similar messages [Mon Jul 2 10:46:09 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 10:46:09 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) Skipped 1879628 previous similar messages [Mon Jul 2 10:46:09 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 10:46:09 2018] Lustre: Skipped 1879627 previous similar messages [Mon Jul 2 10:46:09 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 10:46:09 2018] Lustre: Skipped 1879627 previous similar messages [Mon Jul 2 10:56:09 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 10:56:09 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) Skipped 1418649 previous similar messages [Mon Jul 2 10:56:09 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 10:56:09 2018] Lustre: Skipped 1418656 previous similar messages [Mon Jul 2 10:56:09 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 10:56:09 2018] Lustre: Skipped 1418656 previous similar messages [Mon Jul 2 11:06:09 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 11:06:09 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) Skipped 1510025 previous similar messages [Mon Jul 2 11:06:09 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 11:06:09 2018] Lustre: Skipped 1510005 previous similar messages [Mon Jul 2 11:06:09 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 11:06:09 2018] Lustre: Skipped 1510006 previous similar messages [Mon Jul 2 11:17:34 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 11:17:34 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) Skipped 2349523 previous similar messages [Mon Jul 2 11:17:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 11:17:34 2018] Lustre: Skipped 2349494 previous similar messages [Mon Jul 2 11:17:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 11:17:34 2018] Lustre: Skipped 2349495 previous similar messages [Mon Jul 2 11:27:35 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 11:27:35 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 1925153 previous similar messages [Mon Jul 2 11:27:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 11:27:35 2018] Lustre: Skipped 1925154 previous similar messages [Mon Jul 2 11:27:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 11:27:35 2018] Lustre: Skipped 1925154 previous similar messages [Mon Jul 2 11:37:01 2018] LustreError: 39542:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff881d962f4500 x1601882372813680/t0(0) o37->b7a00d74-04ff-e9ba-691c-377936bcb774@172.16.230.91@o2ib:746/0 lens 568/440 e 0 to 0 dl 1530495221 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 11:37:01 2018] LustreError: 73132:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff881f3562c850 x1601882372813680/t0(0) o37->b7a00d74-04ff-e9ba-691c-377936bcb774@172.16.230.91@o2ib:746/0 lens 568/440 e 0 to 0 dl 1530495221 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 11:37:01 2018] LustreError: 73132:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 17 previous similar messages [Mon Jul 2 11:37:01 2018] LustreError: 39542:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 7 previous similar messages [Mon Jul 2 11:37:12 2018] LustreError: 39542:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff881911b10300 x1601882373632704/t0(0) o37->b7a00d74-04ff-e9ba-691c-377936bcb774@172.16.230.91@o2ib:2/0 lens 568/440 e 0 to 0 dl 1530495232 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 11:37:12 2018] LustreError: 73132:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff881f3562a050 x1601882373632704/t0(0) o37->b7a00d74-04ff-e9ba-691c-377936bcb774@172.16.230.91@o2ib:2/0 lens 568/440 e 0 to 0 dl 1530495232 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 11:37:12 2018] LNet: 2859:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.230.91@o2ib [Mon Jul 2 11:37:12 2018] LNet: 2859:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 8 previous similar messages [Mon Jul 2 11:37:35 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 11:37:35 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) Skipped 1909085 previous similar messages [Mon Jul 2 11:37:35 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 11:37:35 2018] Lustre: Skipped 1909084 previous similar messages [Mon Jul 2 11:37:35 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 11:37:35 2018] Lustre: Skipped 1909084 previous similar messages [Mon Jul 2 11:48:34 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 11:48:34 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) Skipped 2221576 previous similar messages [Mon Jul 2 11:48:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 11:48:34 2018] Lustre: Skipped 2221576 previous similar messages [Mon Jul 2 11:48:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 11:48:34 2018] Lustre: Skipped 2221576 previous similar messages [Mon Jul 2 11:58:27 2018] igb 0000:01:00.0 em1: igb: em1 NIC Link is Down [Mon Jul 2 11:58:32 2018] sd 1:0:1:1: Inquiry data has changed [Mon Jul 2 11:58:34 2018] LustreError: 146455:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 11:58:34 2018] LustreError: 146455:0:(lod_dev.c:1414:lod_sync()) Skipped 1431302 previous similar messages [Mon Jul 2 11:58:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 11:58:34 2018] Lustre: Skipped 1431306 previous similar messages [Mon Jul 2 11:58:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 11:58:34 2018] Lustre: Skipped 1431306 previous similar messages [Mon Jul 2 11:58:35 2018] sd 1:0:1:0: Inquiry data has changed [Mon Jul 2 12:08:34 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 12:08:34 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) Skipped 951900 previous similar messages [Mon Jul 2 12:08:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 12:08:34 2018] Lustre: Skipped 952092 previous similar messages [Mon Jul 2 12:08:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 12:08:34 2018] Lustre: Skipped 952092 previous similar messages [Mon Jul 2 12:09:01 2018] igb 0000:01:00.0 em1: igb: em1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [Mon Jul 2 12:10:32 2018] igb 0000:01:00.0 em1: igb: em1 NIC Link is Down [Mon Jul 2 12:18:34 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 12:18:34 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 2429451 previous similar messages [Mon Jul 2 12:18:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 12:18:34 2018] Lustre: Skipped 2429488 previous similar messages [Mon Jul 2 12:18:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 12:18:34 2018] Lustre: Skipped 2429488 previous similar messages [Mon Jul 2 12:28:34 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 12:28:34 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) Skipped 2408796 previous similar messages [Mon Jul 2 12:28:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 12:28:34 2018] Lustre: Skipped 2408816 previous similar messages [Mon Jul 2 12:28:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 12:28:34 2018] Lustre: Skipped 2408816 previous similar messages [Mon Jul 2 12:33:21 2018] igb 0000:01:00.0 em1: igb: em1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [Mon Jul 2 12:33:28 2018] sd 1:0:1:1: Inquiry data has changed [Mon Jul 2 12:33:31 2018] sd 1:0:1:0: Inquiry data has changed [Mon Jul 2 12:33:39 2018] sd 1:0:0:0: Inquiry data has changed [Mon Jul 2 12:33:39 2018] sd 1:0:0:1: Inquiry data has changed [Mon Jul 2 12:38:34 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 12:38:34 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 2382349 previous similar messages [Mon Jul 2 12:38:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 12:38:34 2018] Lustre: Skipped 2382325 previous similar messages [Mon Jul 2 12:38:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 12:38:34 2018] Lustre: Skipped 2382325 previous similar messages [Mon Jul 2 12:48:34 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 12:48:34 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 2431577 previous similar messages [Mon Jul 2 12:48:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 12:48:34 2018] Lustre: Skipped 2431578 previous similar messages [Mon Jul 2 12:48:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 12:48:34 2018] Lustre: Skipped 2431578 previous similar messages [Mon Jul 2 12:58:34 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 12:58:34 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) Skipped 2386778 previous similar messages [Mon Jul 2 12:58:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 12:58:34 2018] Lustre: Skipped 2386779 previous similar messages [Mon Jul 2 12:58:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 12:58:34 2018] Lustre: Skipped 2386779 previous similar messages [Mon Jul 2 13:08:34 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 13:08:34 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 2408420 previous similar messages [Mon Jul 2 13:08:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 13:08:34 2018] Lustre: Skipped 2408422 previous similar messages [Mon Jul 2 13:08:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 13:08:34 2018] Lustre: Skipped 2408422 previous similar messages [Mon Jul 2 13:18:34 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 13:18:34 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) Skipped 2382096 previous similar messages [Mon Jul 2 13:18:34 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 13:18:34 2018] Lustre: Skipped 2382107 previous similar messages [Mon Jul 2 13:18:34 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 13:18:34 2018] Lustre: Skipped 2382107 previous similar messages [Mon Jul 2 13:29:09 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 13:29:09 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) Skipped 2114270 previous similar messages [Mon Jul 2 13:29:09 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 13:29:09 2018] Lustre: Skipped 2114026 previous similar messages [Mon Jul 2 13:29:09 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 13:29:09 2018] Lustre: Skipped 2114026 previous similar messages [Mon Jul 2 13:39:52 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 13:39:52 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) Skipped 1641517 previous similar messages [Mon Jul 2 13:39:52 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 13:39:52 2018] Lustre: Skipped 1641517 previous similar messages [Mon Jul 2 13:39:52 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 13:39:52 2018] Lustre: Skipped 1641517 previous similar messages [Mon Jul 2 13:49:52 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 13:49:52 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) Skipped 1915494 previous similar messages [Mon Jul 2 13:49:52 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 13:49:52 2018] Lustre: Skipped 1915495 previous similar messages [Mon Jul 2 13:49:52 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 13:49:52 2018] Lustre: Skipped 1915495 previous similar messages [Mon Jul 2 14:00:58 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 14:00:58 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) Skipped 1258741 previous similar messages [Mon Jul 2 14:00:58 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 14:00:58 2018] Lustre: Skipped 1258740 previous similar messages [Mon Jul 2 14:00:58 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 14:00:58 2018] Lustre: Skipped 1258740 previous similar messages [Mon Jul 2 14:10:58 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 14:10:58 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) Skipped 965366 previous similar messages [Mon Jul 2 14:10:58 2018] Lustre: lustre-MDT0000: Client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) reconnecting [Mon Jul 2 14:10:58 2018] Lustre: Skipped 965441 previous similar messages [Mon Jul 2 14:10:58 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Mon Jul 2 14:10:58 2018] Lustre: Skipped 965441 previous similar messages [Mon Jul 2 14:13:59 2018] LNetError: 53942:0:(o2iblnd_cb.c:2862:kiblnd_rejected()) 172.16.230.91@o2ib rejected: o2iblnd fatal error [Mon Jul 2 14:16:09 2018] Lustre: lustre-MDT0000: haven't heard from client b7a00d74-04ff-e9ba-691c-377936bcb774 (at 172.16.230.91@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8820225f9000, cur 1530504763 expire 1530504613 last 1530504536 [Mon Jul 2 14:22:05 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 14:22:05 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 383020 previous similar messages [Mon Jul 2 14:22:05 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 14:22:05 2018] Lustre: Skipped 382944 previous similar messages [Mon Jul 2 14:22:05 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 14:22:05 2018] Lustre: Skipped 382944 previous similar messages [Mon Jul 2 14:33:21 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 14:33:21 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 1264456 previous similar messages [Mon Jul 2 14:33:21 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 14:33:21 2018] Lustre: Skipped 1264456 previous similar messages [Mon Jul 2 14:33:21 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 14:33:21 2018] Lustre: Skipped 1264456 previous similar messages [Mon Jul 2 14:44:58 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 14:44:58 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 439215 previous similar messages [Mon Jul 2 14:44:58 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 14:44:58 2018] Lustre: Skipped 439215 previous similar messages [Mon Jul 2 14:44:58 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 14:44:58 2018] Lustre: Skipped 439217 previous similar messages [Mon Jul 2 14:55:55 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 14:55:55 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 1683478 previous similar messages [Mon Jul 2 14:55:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 14:55:55 2018] Lustre: Skipped 1683478 previous similar messages [Mon Jul 2 14:55:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 14:55:55 2018] Lustre: Skipped 1683478 previous similar messages [Mon Jul 2 14:58:00 2018] LNetError: 2851:0:(o2iblnd_cb.c:3251:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Mon Jul 2 14:58:00 2018] LNetError: 2851:0:(o2iblnd_cb.c:3326:kiblnd_check_conns()) Timed out RDMA with 172.16.230.91@o2ib (162): c: 6, oc: 0, rc: 8 [Mon Jul 2 14:59:05 2018] Lustre: MGS: haven't heard from client a8dbbc22-26fc-204d-21bd-d2ec7877a942 (at 172.16.230.91@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d1034d400, cur 1530507339 expire 1530507189 last 1530507112 [Mon Jul 2 15:05:55 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 15:05:55 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 929390 previous similar messages [Mon Jul 2 15:05:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 15:05:55 2018] Lustre: Skipped 929390 previous similar messages [Mon Jul 2 15:05:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 15:05:55 2018] Lustre: Skipped 929390 previous similar messages [Mon Jul 2 15:16:09 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 15:16:09 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 1485422 previous similar messages [Mon Jul 2 15:16:09 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 15:16:09 2018] Lustre: Skipped 1485422 previous similar messages [Mon Jul 2 15:16:09 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 15:16:09 2018] Lustre: Skipped 1485422 previous similar messages [Mon Jul 2 15:27:04 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 15:27:04 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 251844 previous similar messages [Mon Jul 2 15:27:04 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 15:27:04 2018] Lustre: Skipped 251844 previous similar messages [Mon Jul 2 15:27:04 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 15:27:04 2018] Lustre: Skipped 251844 previous similar messages [Mon Jul 2 15:37:04 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 15:37:04 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 2313988 previous similar messages [Mon Jul 2 15:37:04 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 15:37:04 2018] Lustre: Skipped 2313997 previous similar messages [Mon Jul 2 15:37:04 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 15:37:04 2018] Lustre: Skipped 2313997 previous similar messages [Mon Jul 2 15:47:04 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 15:47:04 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 2319362 previous similar messages [Mon Jul 2 15:47:04 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 15:47:04 2018] Lustre: Skipped 2319358 previous similar messages [Mon Jul 2 15:47:04 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 15:47:04 2018] Lustre: Skipped 2319358 previous similar messages [Mon Jul 2 15:57:04 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 15:57:04 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2342199 previous similar messages [Mon Jul 2 15:57:04 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 15:57:04 2018] Lustre: Skipped 2342205 previous similar messages [Mon Jul 2 15:57:04 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 15:57:04 2018] Lustre: Skipped 2342205 previous similar messages [Mon Jul 2 16:07:04 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 16:07:04 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 2309135 previous similar messages [Mon Jul 2 16:07:04 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 16:07:04 2018] Lustre: Skipped 2309133 previous similar messages [Mon Jul 2 16:07:04 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 16:07:04 2018] Lustre: Skipped 2309133 previous similar messages [Mon Jul 2 16:17:04 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 16:17:04 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 2311297 previous similar messages [Mon Jul 2 16:17:04 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 16:17:04 2018] Lustre: Skipped 2311297 previous similar messages [Mon Jul 2 16:17:04 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 16:17:04 2018] Lustre: Skipped 2311297 previous similar messages [Mon Jul 2 16:27:04 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 16:27:04 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 1883171 previous similar messages [Mon Jul 2 16:27:04 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 16:27:04 2018] Lustre: Skipped 1883172 previous similar messages [Mon Jul 2 16:27:04 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 16:27:04 2018] Lustre: Skipped 1883174 previous similar messages [Mon Jul 2 16:37:04 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 16:37:04 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 1385667 previous similar messages [Mon Jul 2 16:37:04 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 16:37:04 2018] Lustre: Skipped 1385666 previous similar messages [Mon Jul 2 16:37:04 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 16:37:04 2018] Lustre: Skipped 1385666 previous similar messages [Mon Jul 2 16:44:14 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2f9050 x1604829785197520/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:303/0 lens 568/440 e 0 to 0 dl 1530513653 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 16:44:14 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8805b178d100 x1604829785197520/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:303/0 lens 568/440 e 0 to 0 dl 1530513653 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 16:44:14 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 3 previous similar messages [Mon Jul 2 16:44:14 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Mon Jul 2 16:44:15 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 16:44:17 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff88023f79e000 x1604829785511664/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:306/0 lens 568/440 e 0 to 0 dl 1530513656 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 16:44:17 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88023f79e300 x1604829785511664/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:306/0 lens 568/440 e 0 to 0 dl 1530513656 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 16:44:17 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 2 previous similar messages [Mon Jul 2 16:44:17 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 16:44:17 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Mon Jul 2 16:44:20 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 16:44:24 2018] LustreError: 73115:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8805dfb33900 x1604829786269040/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:313/0 lens 568/440 e 0 to 0 dl 1530513663 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 16:44:24 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880111dd0f00 x1604829786269040/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:313/0 lens 568/440 e 0 to 0 dl 1530513663 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 16:44:24 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 3 previous similar messages [Mon Jul 2 16:44:24 2018] LustreError: 73115:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Mon Jul 2 16:44:35 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880043e29800 x1604829787432688/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:324/0 lens 568/440 e 0 to 0 dl 1530513674 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 16:44:35 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880119ef8600 x1604829787432688/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:324/0 lens 568/440 e 0 to 0 dl 1530513674 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 16:44:35 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 2 previous similar messages [Mon Jul 2 16:44:35 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 16:44:35 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 1 previous similar message [Mon Jul 2 16:44:35 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 2 previous similar messages [Mon Jul 2 16:44:50 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 16:44:50 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 1 previous similar message [Mon Jul 2 16:44:59 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8809790fe900 x1604829790069632/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:348/0 lens 568/440 e 0 to 0 dl 1530513698 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 16:44:59 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8804ffb69800 x1604829790069632/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:348/0 lens 568/440 e 0 to 0 dl 1530513698 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 16:44:59 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 5 previous similar messages [Mon Jul 2 16:44:59 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 2 previous similar messages [Mon Jul 2 16:46:43 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88066bff9500 x1604829801410736/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:452/0 lens 568/440 e 0 to 0 dl 1530513802 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 16:46:43 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 16:46:43 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 1 previous similar message [Mon Jul 2 16:46:43 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 2 previous similar messages [Mon Jul 2 16:46:49 2018] LustreError: 73115:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff88003ebe2700 x1604829802056320/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:458/0 lens 568/440 e 0 to 0 dl 1530513808 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 16:47:04 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 16:47:04 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 3898597 previous similar messages [Mon Jul 2 16:47:04 2018] Lustre: lustre-MDT0000: Client c40fc503-ed37-3804-3597-10b39ac4ad1a (at 172.16.229.39@o2ib) reconnecting [Mon Jul 2 16:47:04 2018] Lustre: Skipped 3898930 previous similar messages [Mon Jul 2 16:47:04 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Mon Jul 2 16:47:04 2018] Lustre: Skipped 3898614 previous similar messages [Mon Jul 2 16:47:28 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 16:47:28 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 7 previous similar messages [Mon Jul 2 16:57:04 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 16:57:04 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 6868047 previous similar messages [Mon Jul 2 16:57:04 2018] Lustre: lustre-MDT0000: Client c40fc503-ed37-3804-3597-10b39ac4ad1a (at 172.16.229.39@o2ib) reconnecting [Mon Jul 2 16:57:04 2018] Lustre: Skipped 6867735 previous similar messages [Mon Jul 2 16:57:04 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Mon Jul 2 16:57:04 2018] Lustre: Skipped 6866534 previous similar messages [Mon Jul 2 17:04:01 2018] LustreError: 73115:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff88003eb51800 x1604829914932864/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:735/0 lens 568/440 e 0 to 0 dl 1530514840 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 17:04:01 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880966f8f800 x1604829914932864/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:735/0 lens 568/440 e 0 to 0 dl 1530514840 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 17:04:01 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 18 previous similar messages [Mon Jul 2 17:04:01 2018] LustreError: 73115:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 6 previous similar messages [Mon Jul 2 17:04:04 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 17:04:11 2018] LustreError: 73115:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880966f8c500 x1604829916040112/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:745/0 lens 568/440 e 0 to 0 dl 1530514850 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 17:04:11 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880b5ce20300 x1604829916040112/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:745/0 lens 568/440 e 0 to 0 dl 1530514850 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 17:04:11 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 6 previous similar messages [Mon Jul 2 17:04:11 2018] LustreError: 73115:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 3 previous similar messages [Mon Jul 2 17:04:16 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 17:04:16 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 2 previous similar messages [Mon Jul 2 17:04:30 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffe312050 x1604829918001872/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:45/0 lens 568/440 e 0 to 0 dl 1530514905 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 17:04:30 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 6 previous similar messages [Mon Jul 2 17:04:37 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880bf5ef4200 x1604829918898112/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:53/0 lens 568/440 e 0 to 0 dl 1530514913 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 17:04:37 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 17:04:37 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 2 previous similar messages [Mon Jul 2 17:04:37 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 3 previous similar messages [Mon Jul 2 17:05:10 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880a222ef800 x1604829922659440/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:86/0 lens 568/440 e 0 to 0 dl 1530514946 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 17:05:10 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 17 previous similar messages [Mon Jul 2 17:05:17 2018] LustreError: 73115:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe2f6450 x1604829923419616/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:93/0 lens 568/440 e 0 to 0 dl 1530514953 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 17:05:17 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 17:05:17 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 8 previous similar messages [Mon Jul 2 17:05:17 2018] LustreError: 73115:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 10 previous similar messages [Mon Jul 2 17:06:26 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880de5fc8f00 x1604829931070272/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:162/0 lens 568/440 e 0 to 0 dl 1530515022 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 17:06:26 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 43 previous similar messages [Mon Jul 2 17:06:35 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff88076c83a400 x1604829932076192/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:171/0 lens 568/440 e 0 to 0 dl 1530515031 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 17:06:35 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 17 previous similar messages [Mon Jul 2 17:07:04 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 17:07:04 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 6888826 previous similar messages [Mon Jul 2 17:07:04 2018] Lustre: lustre-MDT0000: Client c40fc503-ed37-3804-3597-10b39ac4ad1a (at 172.16.229.39@o2ib) reconnecting [Mon Jul 2 17:07:04 2018] Lustre: Skipped 6888549 previous similar messages [Mon Jul 2 17:07:04 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Mon Jul 2 17:07:04 2018] Lustre: Skipped 6887423 previous similar messages [Mon Jul 2 17:08:14 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 17:08:14 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 19 previous similar messages [Mon Jul 2 17:08:59 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880a5f326f00 x1604829947916208/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:315/0 lens 568/440 e 0 to 0 dl 1530515175 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 17:08:59 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 20 previous similar messages [Mon Jul 2 17:13:53 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe2f7050 x1604829980041168/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:609/0 lens 568/440 e 0 to 0 dl 1530515469 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 17:13:53 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 17:13:53 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 6 previous similar messages [Mon Jul 2 17:13:53 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 11 previous similar messages [Mon Jul 2 17:13:59 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880119ef9800 x1604829980714160/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:578/0 lens 568/440 e 0 to 0 dl 1530515438 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 17:13:59 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 8 previous similar messages [Mon Jul 2 17:17:04 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 17:17:04 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 6537492 previous similar messages [Mon Jul 2 17:17:04 2018] Lustre: lustre-MDT0000: Client c40fc503-ed37-3804-3597-10b39ac4ad1a (at 172.16.229.39@o2ib) reconnecting [Mon Jul 2 17:17:04 2018] Lustre: Skipped 6537119 previous similar messages [Mon Jul 2 17:17:04 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Mon Jul 2 17:17:04 2018] Lustre: Skipped 6535870 previous similar messages [Mon Jul 2 17:27:04 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 17:27:04 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 6025147 previous similar messages [Mon Jul 2 17:27:04 2018] Lustre: lustre-MDT0000: Client c40fc503-ed37-3804-3597-10b39ac4ad1a (at 172.16.229.39@o2ib) reconnecting [Mon Jul 2 17:27:04 2018] Lustre: Skipped 6025149 previous similar messages [Mon Jul 2 17:27:04 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 17:27:04 2018] Lustre: Skipped 6024144 previous similar messages [Mon Jul 2 17:36:25 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe311c50 x1604830110713504/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:413/0 lens 568/440 e 0 to 0 dl 1530516783 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 17:36:25 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ff3a28600 x1604830110713504/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:413/0 lens 568/440 e 0 to 0 dl 1530516783 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 17:36:25 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 33 previous similar messages [Mon Jul 2 17:36:25 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 21 previous similar messages [Mon Jul 2 17:36:26 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 17:36:26 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 19 previous similar messages [Mon Jul 2 17:37:04 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 17:37:04 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 5735542 previous similar messages [Mon Jul 2 17:37:04 2018] Lustre: lustre-MDT0000: Client c40fc503-ed37-3804-3597-10b39ac4ad1a (at 172.16.229.39@o2ib) reconnecting [Mon Jul 2 17:37:04 2018] Lustre: Skipped 5735328 previous similar messages [Mon Jul 2 17:37:04 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Mon Jul 2 17:37:04 2018] Lustre: Skipped 5734408 previous similar messages [Mon Jul 2 17:37:10 2018] LustreError: 73115:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880111dd3300 x1604830115656848/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:458/0 lens 568/440 e 0 to 0 dl 1530516828 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 17:37:10 2018] LustreError: 73115:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 82 previous similar messages [Mon Jul 2 17:37:16 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 17:37:16 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 7 previous similar messages [Mon Jul 2 17:45:49 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880408b68c00 x1604830162466656/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:222/0 lens 568/440 e 0 to 0 dl 1530517347 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 17:45:49 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 17:45:49 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 4 previous similar messages [Mon Jul 2 17:45:49 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 27 previous similar messages [Mon Jul 2 17:45:50 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe317c50 x1604830162578864/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:223/0 lens 568/440 e 0 to 0 dl 1530517348 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 17:45:50 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 43 previous similar messages [Mon Jul 2 17:47:04 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 17:47:04 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 5591935 previous similar messages [Mon Jul 2 17:47:04 2018] Lustre: lustre-MDT0000: Client c40fc503-ed37-3804-3597-10b39ac4ad1a (at 172.16.229.39@o2ib) reconnecting [Mon Jul 2 17:47:04 2018] Lustre: Skipped 5591496 previous similar messages [Mon Jul 2 17:47:04 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Mon Jul 2 17:47:04 2018] Lustre: Skipped 5590926 previous similar messages [Mon Jul 2 17:57:04 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 17:57:04 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 6245410 previous similar messages [Mon Jul 2 17:57:04 2018] Lustre: lustre-MDT0000: Client c40fc503-ed37-3804-3597-10b39ac4ad1a (at 172.16.229.39@o2ib) reconnecting [Mon Jul 2 17:57:04 2018] Lustre: Skipped 6245177 previous similar messages [Mon Jul 2 17:57:04 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Mon Jul 2 17:57:04 2018] Lustre: Skipped 6244383 previous similar messages [Mon Jul 2 18:07:04 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 18:07:04 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 6052995 previous similar messages [Mon Jul 2 18:07:04 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Mon Jul 2 18:07:04 2018] Lustre: Skipped 6052342 previous similar messages [Mon Jul 2 18:07:04 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 18:07:04 2018] Lustre: Skipped 6053005 previous similar messages [Mon Jul 2 18:07:44 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2fc050 x1604830300238656/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:28/0 lens 568/440 e 0 to 0 dl 1530518663 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 18:07:44 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880a305ab900 x1604830300238656/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:28/0 lens 568/440 e 0 to 0 dl 1530518663 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 18:07:44 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 19 previous similar messages [Mon Jul 2 18:07:44 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 18:07:44 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 8 previous similar messages [Mon Jul 2 18:07:44 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 15 previous similar messages [Mon Jul 2 18:08:08 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8804932fd100 x1604830302836144/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:51/0 lens 568/440 e 0 to 0 dl 1530518686 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 18:08:08 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8809d1079200 x1604830302836144/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:51/0 lens 568/440 e 0 to 0 dl 1530518686 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 18:08:08 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 13 previous similar messages [Mon Jul 2 18:08:08 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 18:08:08 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 6 previous similar messages [Mon Jul 2 18:08:08 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 6 previous similar messages [Mon Jul 2 18:17:04 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 18:17:04 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 6785330 previous similar messages [Mon Jul 2 18:17:04 2018] Lustre: lustre-MDT0000: Client c40fc503-ed37-3804-3597-10b39ac4ad1a (at 172.16.229.39@o2ib) reconnecting [Mon Jul 2 18:17:04 2018] Lustre: Skipped 6785048 previous similar messages [Mon Jul 2 18:17:04 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Mon Jul 2 18:17:04 2018] Lustre: Skipped 6784117 previous similar messages [Mon Jul 2 18:27:04 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 18:27:04 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 6895095 previous similar messages [Mon Jul 2 18:27:04 2018] Lustre: lustre-MDT0000: Client c40fc503-ed37-3804-3597-10b39ac4ad1a (at 172.16.229.39@o2ib) reconnecting [Mon Jul 2 18:27:04 2018] Lustre: Skipped 6894731 previous similar messages [Mon Jul 2 18:27:04 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Mon Jul 2 18:27:04 2018] Lustre: Skipped 6893724 previous similar messages [Mon Jul 2 18:28:17 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffe317c50 x1604830434930016/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:506/0 lens 568/440 e 0 to 0 dl 1530519896 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 18:28:17 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 18:28:17 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 2 previous similar messages [Mon Jul 2 18:28:17 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 8 previous similar messages [Mon Jul 2 18:28:20 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880326a14500 x1604830435260128/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:509/0 lens 568/440 e 0 to 0 dl 1530519899 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 18:28:20 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 6 previous similar messages [Mon Jul 2 18:28:23 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8803fd650c00 x1604830435589024/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:512/0 lens 568/440 e 0 to 0 dl 1530519902 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 18:28:23 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 18:28:23 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 2 previous similar messages [Mon Jul 2 18:28:23 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 6 previous similar messages [Mon Jul 2 18:28:27 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe314450 x1604830436028400/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:516/0 lens 568/440 e 0 to 0 dl 1530519906 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 18:28:27 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Mon Jul 2 18:28:33 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88043877ad00 x1604830436561136/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:521/0 lens 568/440 e 0 to 0 dl 1530519911 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 18:28:33 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 4 previous similar messages [Mon Jul 2 18:28:38 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe316850 x1604830437196208/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:527/0 lens 568/440 e 0 to 0 dl 1530519917 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 18:28:38 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 18:28:38 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 2 previous similar messages [Mon Jul 2 18:28:38 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 2 previous similar messages [Mon Jul 2 18:28:53 2018] LustreError: 73116:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88028f7fbf00 x1604830438856928/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:579/0 lens 568/440 e 0 to 0 dl 1530519969 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 18:28:53 2018] LustreError: 73116:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 4 previous similar messages [Mon Jul 2 18:30:55 2018] LustreError: 73116:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffe2f6050 x1604830451421136/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:700/0 lens 568/440 e 0 to 0 dl 1530520090 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 18:30:55 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 18:30:55 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 1 previous similar message [Mon Jul 2 18:30:55 2018] LustreError: 73116:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 1 previous similar message [Mon Jul 2 18:30:58 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8801199ce000 x1604830451749184/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:703/0 lens 568/440 e 0 to 0 dl 1530520093 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 18:30:58 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Mon Jul 2 18:31:40 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe316050 x1604830456340464/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:745/0 lens 568/440 e 0 to 0 dl 1530520135 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 18:31:40 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 18:31:40 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 8 previous similar messages [Mon Jul 2 18:31:40 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 7 previous similar messages [Mon Jul 2 18:33:40 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2fbc50 x1604830469506112/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:110/0 lens 568/440 e 0 to 0 dl 1530520255 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 18:33:40 2018] LustreError: 73116:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880868bf4e00 x1604830469506112/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:110/0 lens 568/440 e 0 to 0 dl 1530520255 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 18:33:40 2018] LustreError: 73116:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 19 previous similar messages [Mon Jul 2 18:33:40 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 18:37:04 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 18:37:04 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 5642681 previous similar messages [Mon Jul 2 18:37:05 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 18:37:05 2018] Lustre: Skipped 5642481 previous similar messages [Mon Jul 2 18:37:05 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 18:37:05 2018] Lustre: Skipped 5641500 previous similar messages [Mon Jul 2 18:46:45 2018] LustreError: 73116:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880a53db0f00 x1604830543895952/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:140/0 lens 568/440 e 0 to 0 dl 1530521040 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 18:46:45 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffe316850 x1604830543895952/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:140/0 lens 568/440 e 0 to 0 dl 1530521040 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 18:46:45 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 17 previous similar messages [Mon Jul 2 18:46:45 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 18:46:45 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 6 previous similar messages [Mon Jul 2 18:46:45 2018] LustreError: 73116:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 5 previous similar messages [Mon Jul 2 18:47:04 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 18:47:04 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 6848282 previous similar messages [Mon Jul 2 18:47:05 2018] Lustre: lustre-MDT0000: Client c40fc503-ed37-3804-3597-10b39ac4ad1a (at 172.16.229.39@o2ib) reconnecting [Mon Jul 2 18:47:05 2018] Lustre: Skipped 6847917 previous similar messages [Mon Jul 2 18:47:05 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Mon Jul 2 18:47:05 2018] Lustre: Skipped 6846459 previous similar messages [Mon Jul 2 18:51:27 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880324640f00 x1604830575007824/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:422/0 lens 568/440 e 0 to 0 dl 1530521322 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 18:51:27 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880324641200 x1604830575007824/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:422/0 lens 568/440 e 0 to 0 dl 1530521322 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 18:51:27 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 9 previous similar messages [Mon Jul 2 18:51:27 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 18:51:27 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 4 previous similar messages [Mon Jul 2 18:51:27 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 5 previous similar messages [Mon Jul 2 18:52:12 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880938e15700 x1604830579976752/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:467/0 lens 568/440 e 0 to 0 dl 1530521367 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 18:52:12 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffd2f9050 x1604830579976752/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:467/0 lens 568/440 e 0 to 0 dl 1530521367 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 18:52:12 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 17 previous similar messages [Mon Jul 2 18:52:12 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 18:52:12 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 8 previous similar messages [Mon Jul 2 18:52:12 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 5 previous similar messages [Mon Jul 2 18:57:05 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 18:57:05 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 6278093 previous similar messages [Mon Jul 2 18:57:05 2018] Lustre: lustre-MDT0000: Client c40fc503-ed37-3804-3597-10b39ac4ad1a (at 172.16.229.39@o2ib) reconnecting [Mon Jul 2 18:57:05 2018] Lustre: Skipped 6277745 previous similar messages [Mon Jul 2 18:57:05 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Mon Jul 2 18:57:05 2018] Lustre: Skipped 6276383 previous similar messages [Mon Jul 2 19:00:08 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe315450 x1604830621159216/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:188/0 lens 568/440 e 0 to 0 dl 1530521843 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 19:00:08 2018] LustreError: 73116:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ff6a33600 x1604830621159216/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:188/0 lens 568/440 e 0 to 0 dl 1530521843 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 19:00:08 2018] LustreError: 73116:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 1 previous similar message [Mon Jul 2 19:00:08 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 19:02:38 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880f045d1b00 x1604830637696912/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:338/0 lens 568/440 e 0 to 0 dl 1530521993 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 19:02:38 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffe311850 x1604830637696912/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:338/0 lens 568/440 e 0 to 0 dl 1530521993 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 19:02:38 2018] LustreError: 73115:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 68 previous similar messages [Mon Jul 2 19:02:38 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 19:02:38 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 33 previous similar messages [Mon Jul 2 19:02:38 2018] LustreError: 159528:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 30 previous similar messages [Mon Jul 2 19:07:05 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 19:07:05 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 6922496 previous similar messages [Mon Jul 2 19:07:05 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 19:07:05 2018] Lustre: Skipped 6922150 previous similar messages [Mon Jul 2 19:07:05 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 19:07:05 2018] Lustre: Skipped 6920932 previous similar messages [Mon Jul 2 19:17:05 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 19:17:05 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 6912779 previous similar messages [Mon Jul 2 19:17:05 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 19:17:05 2018] Lustre: Skipped 6912403 previous similar messages [Mon Jul 2 19:17:05 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 19:17:05 2018] Lustre: Skipped 6911452 previous similar messages [Mon Jul 2 19:20:21 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2f9450 x1604830755286720/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:646/0 lens 568/440 e 0 to 0 dl 1530523056 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 19:20:21 2018] LustreError: 73116:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8807e02e8300 x1604830755286720/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:646/0 lens 568/440 e 0 to 0 dl 1530523056 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 19:20:21 2018] LustreError: 73116:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 29 previous similar messages [Mon Jul 2 19:20:21 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 19:20:21 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 14 previous similar messages [Mon Jul 2 19:20:21 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 11 previous similar messages [Mon Jul 2 19:21:06 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880c05e2bf00 x1604830760272976/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:654/0 lens 568/440 e 0 to 0 dl 1530523064 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 19:21:06 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880448f3da00 x1604830760272976/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:654/0 lens 568/440 e 0 to 0 dl 1530523064 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 19:21:06 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 17 previous similar messages [Mon Jul 2 19:21:06 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 19:21:06 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 7 previous similar messages [Mon Jul 2 19:21:06 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 7 previous similar messages [Mon Jul 2 19:22:27 2018] LustreError: 73116:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880256d6e900 x1604830769223696/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:17/0 lens 568/440 e 0 to 0 dl 1530523182 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 19:22:27 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880256d6ef00 x1604830769223696/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:17/0 lens 568/440 e 0 to 0 dl 1530523182 ref 1 fl Interpret:/2/0 rc 0/0 [Mon Jul 2 19:22:27 2018] LustreError: 159528:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 38 previous similar messages [Mon Jul 2 19:22:27 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Mon Jul 2 19:22:27 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 16 previous similar messages [Mon Jul 2 19:22:27 2018] LustreError: 73116:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 10 previous similar messages [Mon Jul 2 19:27:05 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 19:27:05 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 6907306 previous similar messages [Mon Jul 2 19:27:05 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Mon Jul 2 19:27:05 2018] Lustre: Skipped 6905760 previous similar messages [Mon Jul 2 19:27:05 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 19:27:05 2018] Lustre: Skipped 6906921 previous similar messages [Mon Jul 2 19:37:05 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 19:37:05 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 6899040 previous similar messages [Mon Jul 2 19:37:05 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 19:37:05 2018] Lustre: Skipped 6898684 previous similar messages [Mon Jul 2 19:37:05 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 19:37:05 2018] Lustre: Skipped 6897568 previous similar messages [Mon Jul 2 19:41:40 2018] LustreError: 73115:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880829a11800 x1604830896752208/t0(0) o37->c40fc503-ed37-3804-3597-10b39ac4ad1a@172.16.229.39@o2ib:378/0 lens 568/440 e 0 to 0 dl 1530524298 ref 1 fl Interpret:/0/0 rc 0/0 [Mon Jul 2 19:41:40 2018] LustreError: 73115:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 12 previous similar messages [Mon Jul 2 19:43:04 2018] LNet: 161861:0:(o2iblnd_cb.c:2502:kiblnd_passive_connect()) Conn stale 172.16.229.39@o2ib version 12/12 incarnation 1530484672662758/1530524375553037 [Mon Jul 2 19:45:26 2018] Lustre: MGS: haven't heard from client 8aded177-06ee-6037-a661-7a7515050a9f (at 172.16.229.39@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ddd660400, cur 1530524518 expire 1530524368 last 1530524291 [Mon Jul 2 19:45:26 2018] Lustre: Skipped 1 previous similar message [Mon Jul 2 19:45:28 2018] Lustre: lustre-MDT0000: haven't heard from client c40fc503-ed37-3804-3597-10b39ac4ad1a (at 172.16.229.39@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff8820228b5c00, cur 1530524520 expire 1530524370 last 1530524292 [Mon Jul 2 19:47:05 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 19:47:05 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 4198480 previous similar messages [Mon Jul 2 19:47:05 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 19:47:05 2018] Lustre: Skipped 4197940 previous similar messages [Mon Jul 2 19:47:05 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 19:47:05 2018] Lustre: Skipped 4197275 previous similar messages [Mon Jul 2 19:57:05 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 19:57:05 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 2342624 previous similar messages [Mon Jul 2 19:57:05 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 19:57:05 2018] Lustre: Skipped 2342625 previous similar messages [Mon Jul 2 19:57:05 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 19:57:05 2018] Lustre: Skipped 2342625 previous similar messages [Mon Jul 2 20:07:05 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 20:07:05 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 2336039 previous similar messages [Mon Jul 2 20:07:05 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 20:07:05 2018] Lustre: Skipped 2336039 previous similar messages [Mon Jul 2 20:07:05 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 20:07:05 2018] Lustre: Skipped 2336039 previous similar messages [Mon Jul 2 20:17:05 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 20:17:05 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 2263547 previous similar messages [Mon Jul 2 20:17:05 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 20:17:05 2018] Lustre: Skipped 2263520 previous similar messages [Mon Jul 2 20:17:05 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 20:17:05 2018] Lustre: Skipped 2263520 previous similar messages [Mon Jul 2 20:27:05 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 20:27:05 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 2315372 previous similar messages [Mon Jul 2 20:27:05 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 20:27:05 2018] Lustre: Skipped 2315395 previous similar messages [Mon Jul 2 20:27:05 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 20:27:05 2018] Lustre: Skipped 2315395 previous similar messages [Mon Jul 2 20:37:43 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 20:37:43 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 2038966 previous similar messages [Mon Jul 2 20:37:43 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 20:37:43 2018] Lustre: Skipped 2038691 previous similar messages [Mon Jul 2 20:37:43 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 20:37:43 2018] Lustre: Skipped 2038691 previous similar messages [Mon Jul 2 20:47:43 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 20:47:43 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 1834142 previous similar messages [Mon Jul 2 20:47:43 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 20:47:43 2018] Lustre: Skipped 1834142 previous similar messages [Mon Jul 2 20:47:43 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 20:47:43 2018] Lustre: Skipped 1834142 previous similar messages [Mon Jul 2 20:57:44 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 20:57:44 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 1386120 previous similar messages [Mon Jul 2 20:57:44 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 20:57:44 2018] Lustre: Skipped 1386120 previous similar messages [Mon Jul 2 20:57:44 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 20:57:44 2018] Lustre: Skipped 1386120 previous similar messages [Mon Jul 2 21:07:44 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 21:07:44 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 903957 previous similar messages [Mon Jul 2 21:07:44 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 21:07:44 2018] Lustre: Skipped 903957 previous similar messages [Mon Jul 2 21:07:44 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 21:07:44 2018] Lustre: Skipped 903957 previous similar messages [Mon Jul 2 21:17:44 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 21:17:44 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 1343759 previous similar messages [Mon Jul 2 21:17:44 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 21:17:44 2018] Lustre: Skipped 1343759 previous similar messages [Mon Jul 2 21:17:44 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 21:17:44 2018] Lustre: Skipped 1343759 previous similar messages [Mon Jul 2 21:27:44 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 21:27:44 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2244213 previous similar messages [Mon Jul 2 21:27:44 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 21:27:44 2018] Lustre: Skipped 2244213 previous similar messages [Mon Jul 2 21:27:44 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 21:27:44 2018] Lustre: Skipped 2244213 previous similar messages [Mon Jul 2 21:39:23 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 21:39:23 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 1830411 previous similar messages [Mon Jul 2 21:39:23 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 21:39:23 2018] Lustre: Skipped 1830411 previous similar messages [Mon Jul 2 21:39:23 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 21:39:23 2018] Lustre: Skipped 1830411 previous similar messages [Mon Jul 2 21:49:37 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 21:49:37 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 978302 previous similar messages [Mon Jul 2 21:49:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 21:49:37 2018] Lustre: Skipped 978302 previous similar messages [Mon Jul 2 21:49:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 21:49:37 2018] Lustre: Skipped 978302 previous similar messages [Mon Jul 2 21:59:37 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 21:59:37 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 1380545 previous similar messages [Mon Jul 2 21:59:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 21:59:37 2018] Lustre: Skipped 1380550 previous similar messages [Mon Jul 2 21:59:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 21:59:37 2018] Lustre: Skipped 1380550 previous similar messages [Mon Jul 2 22:09:37 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 22:09:37 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 2312150 previous similar messages [Mon Jul 2 22:09:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 22:09:37 2018] Lustre: Skipped 2312150 previous similar messages [Mon Jul 2 22:09:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 22:09:37 2018] Lustre: Skipped 2312150 previous similar messages [Mon Jul 2 22:19:37 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 22:19:37 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 1817280 previous similar messages [Mon Jul 2 22:19:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 22:19:37 2018] Lustre: Skipped 1817279 previous similar messages [Mon Jul 2 22:19:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 22:19:37 2018] Lustre: Skipped 1817279 previous similar messages [Mon Jul 2 22:29:37 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 22:29:37 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 1394276 previous similar messages [Mon Jul 2 22:29:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 22:29:37 2018] Lustre: Skipped 1394321 previous similar messages [Mon Jul 2 22:29:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 22:29:37 2018] Lustre: Skipped 1394321 previous similar messages [Mon Jul 2 22:39:37 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 22:39:37 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 2278208 previous similar messages [Mon Jul 2 22:39:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 22:39:37 2018] Lustre: Skipped 2278202 previous similar messages [Mon Jul 2 22:39:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 22:39:37 2018] Lustre: Skipped 2278202 previous similar messages [Mon Jul 2 22:49:37 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 22:49:37 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 1354088 previous similar messages [Mon Jul 2 22:49:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 22:49:37 2018] Lustre: Skipped 1354090 previous similar messages [Mon Jul 2 22:49:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 22:49:37 2018] Lustre: Skipped 1354090 previous similar messages [Mon Jul 2 22:59:37 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 22:59:37 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 1806690 previous similar messages [Mon Jul 2 22:59:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 22:59:37 2018] Lustre: Skipped 1806685 previous similar messages [Mon Jul 2 22:59:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 22:59:37 2018] Lustre: Skipped 1806685 previous similar messages [Mon Jul 2 23:09:37 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 23:09:37 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 2281777 previous similar messages [Mon Jul 2 23:09:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 23:09:37 2018] Lustre: Skipped 2281784 previous similar messages [Mon Jul 2 23:09:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 23:09:37 2018] Lustre: Skipped 2281784 previous similar messages [Mon Jul 2 23:19:37 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 23:19:37 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 2301021 previous similar messages [Mon Jul 2 23:19:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 23:19:37 2018] Lustre: Skipped 2301021 previous similar messages [Mon Jul 2 23:19:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 23:19:37 2018] Lustre: Skipped 2301021 previous similar messages [Mon Jul 2 23:29:37 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 23:29:37 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 2255180 previous similar messages [Mon Jul 2 23:29:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 23:29:37 2018] Lustre: Skipped 2255180 previous similar messages [Mon Jul 2 23:29:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 23:29:37 2018] Lustre: Skipped 2255180 previous similar messages [Mon Jul 2 23:39:37 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 23:39:37 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 2273437 previous similar messages [Mon Jul 2 23:39:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 23:39:37 2018] Lustre: Skipped 2273434 previous similar messages [Mon Jul 2 23:39:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 23:39:37 2018] Lustre: Skipped 2273434 previous similar messages [Mon Jul 2 23:49:37 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 23:49:37 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 2293062 previous similar messages [Mon Jul 2 23:49:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 23:49:37 2018] Lustre: Skipped 2293063 previous similar messages [Mon Jul 2 23:49:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 23:49:37 2018] Lustre: Skipped 2293063 previous similar messages [Mon Jul 2 23:59:37 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Mon Jul 2 23:59:37 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 2302790 previous similar messages [Mon Jul 2 23:59:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Mon Jul 2 23:59:37 2018] Lustre: Skipped 2302798 previous similar messages [Mon Jul 2 23:59:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Mon Jul 2 23:59:37 2018] Lustre: Skipped 2302798 previous similar messages [Tue Jul 3 00:09:37 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 00:09:37 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 2301603 previous similar messages [Tue Jul 3 00:09:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 00:09:37 2018] Lustre: Skipped 2301601 previous similar messages [Tue Jul 3 00:09:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 00:09:37 2018] Lustre: Skipped 2301601 previous similar messages [Tue Jul 3 00:19:37 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 00:19:37 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 2286074 previous similar messages [Tue Jul 3 00:19:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 00:19:37 2018] Lustre: Skipped 2286076 previous similar messages [Tue Jul 3 00:19:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 00:19:37 2018] Lustre: Skipped 2286076 previous similar messages [Tue Jul 3 00:29:37 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 00:29:37 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 2310522 previous similar messages [Tue Jul 3 00:29:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 00:29:37 2018] Lustre: Skipped 2310518 previous similar messages [Tue Jul 3 00:29:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 00:29:37 2018] Lustre: Skipped 2310518 previous similar messages [Tue Jul 3 00:39:37 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 00:39:37 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 2294450 previous similar messages [Tue Jul 3 00:39:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 00:39:37 2018] Lustre: Skipped 2294446 previous similar messages [Tue Jul 3 00:39:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 00:39:37 2018] Lustre: Skipped 2294446 previous similar messages [Tue Jul 3 00:49:37 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 00:49:37 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 2328206 previous similar messages [Tue Jul 3 00:49:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 00:49:37 2018] Lustre: Skipped 2328179 previous similar messages [Tue Jul 3 00:49:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 00:49:37 2018] Lustre: Skipped 2328179 previous similar messages [Tue Jul 3 00:59:37 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 00:59:37 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 2335598 previous similar messages [Tue Jul 3 00:59:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 00:59:37 2018] Lustre: Skipped 2335591 previous similar messages [Tue Jul 3 00:59:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 00:59:37 2018] Lustre: Skipped 2335591 previous similar messages [Tue Jul 3 01:09:37 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 01:09:37 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 2302543 previous similar messages [Tue Jul 3 01:09:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 01:09:37 2018] Lustre: Skipped 2302538 previous similar messages [Tue Jul 3 01:09:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 01:09:37 2018] Lustre: Skipped 2302538 previous similar messages [Tue Jul 3 01:19:37 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 01:19:37 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 2297615 previous similar messages [Tue Jul 3 01:19:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 01:19:38 2018] Lustre: Skipped 2297610 previous similar messages [Tue Jul 3 01:19:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 01:19:38 2018] Lustre: Skipped 2297610 previous similar messages [Tue Jul 3 01:29:38 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 01:29:38 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 2396547 previous similar messages [Tue Jul 3 01:29:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 01:29:38 2018] Lustre: Skipped 2396560 previous similar messages [Tue Jul 3 01:29:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 01:29:38 2018] Lustre: Skipped 2396560 previous similar messages [Tue Jul 3 01:39:38 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 01:39:38 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 2333457 previous similar messages [Tue Jul 3 01:39:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 01:39:38 2018] Lustre: Skipped 2333498 previous similar messages [Tue Jul 3 01:39:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 01:39:38 2018] Lustre: Skipped 2333498 previous similar messages [Tue Jul 3 01:49:38 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 01:49:38 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 2335440 previous similar messages [Tue Jul 3 01:49:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 01:49:38 2018] Lustre: Skipped 2335428 previous similar messages [Tue Jul 3 01:49:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 01:49:38 2018] Lustre: Skipped 2335428 previous similar messages [Tue Jul 3 01:59:38 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 01:59:38 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 2321516 previous similar messages [Tue Jul 3 01:59:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 01:59:38 2018] Lustre: Skipped 2321483 previous similar messages [Tue Jul 3 01:59:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 01:59:38 2018] Lustre: Skipped 2321483 previous similar messages [Tue Jul 3 02:09:38 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 02:09:38 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 2292927 previous similar messages [Tue Jul 3 02:09:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 02:09:38 2018] Lustre: Skipped 2292923 previous similar messages [Tue Jul 3 02:09:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 02:09:38 2018] Lustre: Skipped 2292923 previous similar messages [Tue Jul 3 02:19:38 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 02:19:38 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 2455859 previous similar messages [Tue Jul 3 02:19:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 02:19:38 2018] Lustre: Skipped 2455926 previous similar messages [Tue Jul 3 02:19:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 02:19:38 2018] Lustre: Skipped 2455926 previous similar messages [Tue Jul 3 02:29:38 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 02:29:38 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 2308596 previous similar messages [Tue Jul 3 02:29:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 02:29:38 2018] Lustre: Skipped 2308587 previous similar messages [Tue Jul 3 02:29:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 02:29:38 2018] Lustre: Skipped 2308587 previous similar messages [Tue Jul 3 02:39:38 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 02:39:38 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 2313284 previous similar messages [Tue Jul 3 02:39:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 02:39:38 2018] Lustre: Skipped 2313285 previous similar messages [Tue Jul 3 02:39:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 02:39:38 2018] Lustre: Skipped 2313285 previous similar messages [Tue Jul 3 02:49:38 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 02:49:38 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 2333305 previous similar messages [Tue Jul 3 02:49:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 02:49:38 2018] Lustre: Skipped 2333308 previous similar messages [Tue Jul 3 02:49:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 02:49:38 2018] Lustre: Skipped 2333308 previous similar messages [Tue Jul 3 02:59:38 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 02:59:38 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 2108934 previous similar messages [Tue Jul 3 02:59:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 02:59:38 2018] Lustre: Skipped 2108931 previous similar messages [Tue Jul 3 02:59:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 02:59:38 2018] Lustre: Skipped 2108931 previous similar messages [Tue Jul 3 03:09:38 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 03:09:38 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2297306 previous similar messages [Tue Jul 3 03:09:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 03:09:38 2018] Lustre: Skipped 2297305 previous similar messages [Tue Jul 3 03:09:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 03:09:38 2018] Lustre: Skipped 2297305 previous similar messages [Tue Jul 3 03:19:38 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 03:19:38 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 2316934 previous similar messages [Tue Jul 3 03:19:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 03:19:38 2018] Lustre: Skipped 2316934 previous similar messages [Tue Jul 3 03:19:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 03:19:38 2018] Lustre: Skipped 2316934 previous similar messages [Tue Jul 3 03:29:38 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 03:29:38 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2311323 previous similar messages [Tue Jul 3 03:29:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 03:29:38 2018] Lustre: Skipped 2311317 previous similar messages [Tue Jul 3 03:29:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 03:29:38 2018] Lustre: Skipped 2311317 previous similar messages [Tue Jul 3 03:39:38 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 03:39:38 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 2313388 previous similar messages [Tue Jul 3 03:39:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 03:39:38 2018] Lustre: Skipped 2313393 previous similar messages [Tue Jul 3 03:39:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 03:39:38 2018] Lustre: Skipped 2313393 previous similar messages [Tue Jul 3 03:49:38 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 03:49:38 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 2304946 previous similar messages [Tue Jul 3 03:49:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 03:49:38 2018] Lustre: Skipped 2304951 previous similar messages [Tue Jul 3 03:49:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 03:49:38 2018] Lustre: Skipped 2304951 previous similar messages [Tue Jul 3 03:59:38 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 03:59:38 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 2332953 previous similar messages [Tue Jul 3 03:59:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 03:59:38 2018] Lustre: Skipped 2332917 previous similar messages [Tue Jul 3 03:59:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 03:59:38 2018] Lustre: Skipped 2332917 previous similar messages [Tue Jul 3 04:09:38 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 04:09:38 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 2289879 previous similar messages [Tue Jul 3 04:09:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 04:09:38 2018] Lustre: Skipped 2289905 previous similar messages [Tue Jul 3 04:09:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 04:09:38 2018] Lustre: Skipped 2289905 previous similar messages [Tue Jul 3 04:19:38 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 04:19:38 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 2263968 previous similar messages [Tue Jul 3 04:19:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 04:19:38 2018] Lustre: Skipped 2263977 previous similar messages [Tue Jul 3 04:19:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 04:19:38 2018] Lustre: Skipped 2263977 previous similar messages [Tue Jul 3 04:29:38 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 04:29:38 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 2297370 previous similar messages [Tue Jul 3 04:29:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 04:29:38 2018] Lustre: Skipped 2297332 previous similar messages [Tue Jul 3 04:29:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 04:29:38 2018] Lustre: Skipped 2297332 previous similar messages [Tue Jul 3 04:39:38 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 04:39:38 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 2313370 previous similar messages [Tue Jul 3 04:39:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 04:39:38 2018] Lustre: Skipped 2313372 previous similar messages [Tue Jul 3 04:39:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 04:39:38 2018] Lustre: Skipped 2313372 previous similar messages [Tue Jul 3 04:49:38 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 04:49:38 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 2341065 previous similar messages [Tue Jul 3 04:49:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 04:49:38 2018] Lustre: Skipped 2341098 previous similar messages [Tue Jul 3 04:49:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 04:49:38 2018] Lustre: Skipped 2341098 previous similar messages [Tue Jul 3 04:59:38 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 04:59:38 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 2338668 previous similar messages [Tue Jul 3 04:59:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 04:59:39 2018] Lustre: Skipped 2338675 previous similar messages [Tue Jul 3 04:59:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 04:59:39 2018] Lustre: Skipped 2338675 previous similar messages [Tue Jul 3 05:09:39 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 05:09:39 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 2361312 previous similar messages [Tue Jul 3 05:09:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 05:09:39 2018] Lustre: Skipped 2361306 previous similar messages [Tue Jul 3 05:09:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 05:09:39 2018] Lustre: Skipped 2361306 previous similar messages [Tue Jul 3 05:19:39 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 05:19:39 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 2299155 previous similar messages [Tue Jul 3 05:19:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 05:19:39 2018] Lustre: Skipped 2299154 previous similar messages [Tue Jul 3 05:19:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 05:19:39 2018] Lustre: Skipped 2299154 previous similar messages [Tue Jul 3 05:29:39 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 05:29:39 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2265756 previous similar messages [Tue Jul 3 05:29:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 05:29:39 2018] Lustre: Skipped 2265753 previous similar messages [Tue Jul 3 05:29:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 05:29:39 2018] Lustre: Skipped 2265753 previous similar messages [Tue Jul 3 05:39:39 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 05:39:39 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 2309439 previous similar messages [Tue Jul 3 05:39:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 05:39:39 2018] Lustre: Skipped 2309444 previous similar messages [Tue Jul 3 05:39:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 05:39:39 2018] Lustre: Skipped 2309444 previous similar messages [Tue Jul 3 05:49:39 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 05:49:39 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 2285503 previous similar messages [Tue Jul 3 05:49:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 05:49:39 2018] Lustre: Skipped 2285500 previous similar messages [Tue Jul 3 05:49:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 05:49:39 2018] Lustre: Skipped 2285500 previous similar messages [Tue Jul 3 05:59:39 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 05:59:39 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2287263 previous similar messages [Tue Jul 3 05:59:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 05:59:39 2018] Lustre: Skipped 2287267 previous similar messages [Tue Jul 3 05:59:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 05:59:39 2018] Lustre: Skipped 2287267 previous similar messages [Tue Jul 3 06:09:39 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 06:09:39 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 2309421 previous similar messages [Tue Jul 3 06:09:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 06:09:39 2018] Lustre: Skipped 2309426 previous similar messages [Tue Jul 3 06:09:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 06:09:39 2018] Lustre: Skipped 2309426 previous similar messages [Tue Jul 3 06:19:39 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 06:19:39 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 2279417 previous similar messages [Tue Jul 3 06:19:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 06:19:39 2018] Lustre: Skipped 2279405 previous similar messages [Tue Jul 3 06:19:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 06:19:39 2018] Lustre: Skipped 2279405 previous similar messages [Tue Jul 3 06:29:39 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 06:29:39 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 2249042 previous similar messages [Tue Jul 3 06:29:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 06:29:39 2018] Lustre: Skipped 2249048 previous similar messages [Tue Jul 3 06:29:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 06:29:39 2018] Lustre: Skipped 2249048 previous similar messages [Tue Jul 3 06:39:39 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 06:39:39 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 2309175 previous similar messages [Tue Jul 3 06:39:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 06:39:39 2018] Lustre: Skipped 2309114 previous similar messages [Tue Jul 3 06:39:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 06:39:39 2018] Lustre: Skipped 2309114 previous similar messages [Tue Jul 3 06:49:39 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 06:49:39 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 2099620 previous similar messages [Tue Jul 3 06:49:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 06:49:39 2018] Lustre: Skipped 2099686 previous similar messages [Tue Jul 3 06:49:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 06:49:39 2018] Lustre: Skipped 2099686 previous similar messages [Tue Jul 3 06:59:39 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 06:59:39 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 2303185 previous similar messages [Tue Jul 3 06:59:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 06:59:39 2018] Lustre: Skipped 2303192 previous similar messages [Tue Jul 3 06:59:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 06:59:39 2018] Lustre: Skipped 2303192 previous similar messages [Tue Jul 3 07:09:39 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 07:09:39 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 2273419 previous similar messages [Tue Jul 3 07:09:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 07:09:39 2018] Lustre: Skipped 2273443 previous similar messages [Tue Jul 3 07:09:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 07:09:39 2018] Lustre: Skipped 2273443 previous similar messages [Tue Jul 3 07:19:39 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 07:19:39 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 2266322 previous similar messages [Tue Jul 3 07:19:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 07:19:39 2018] Lustre: Skipped 2266293 previous similar messages [Tue Jul 3 07:19:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 07:19:39 2018] Lustre: Skipped 2266293 previous similar messages [Tue Jul 3 07:29:39 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 07:29:39 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 2306415 previous similar messages [Tue Jul 3 07:29:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 07:29:39 2018] Lustre: Skipped 2306422 previous similar messages [Tue Jul 3 07:29:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 07:29:39 2018] Lustre: Skipped 2306422 previous similar messages [Tue Jul 3 07:39:39 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 07:39:39 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 2273600 previous similar messages [Tue Jul 3 07:39:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 07:39:39 2018] Lustre: Skipped 2273597 previous similar messages [Tue Jul 3 07:39:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 07:39:39 2018] Lustre: Skipped 2273597 previous similar messages [Tue Jul 3 07:49:39 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 07:49:39 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 2268914 previous similar messages [Tue Jul 3 07:49:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 07:49:39 2018] Lustre: Skipped 2268920 previous similar messages [Tue Jul 3 07:49:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 07:49:39 2018] Lustre: Skipped 2268920 previous similar messages [Tue Jul 3 07:59:39 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 07:59:39 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 2300017 previous similar messages [Tue Jul 3 07:59:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 07:59:39 2018] Lustre: Skipped 2300012 previous similar messages [Tue Jul 3 07:59:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 07:59:39 2018] Lustre: Skipped 2300012 previous similar messages [Tue Jul 3 08:09:39 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 08:09:39 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 2263526 previous similar messages [Tue Jul 3 08:09:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 08:09:39 2018] Lustre: Skipped 2263521 previous similar messages [Tue Jul 3 08:09:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 08:09:39 2018] Lustre: Skipped 2263521 previous similar messages [Tue Jul 3 08:19:39 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 08:19:39 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 2275001 previous similar messages [Tue Jul 3 08:19:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 08:19:39 2018] Lustre: Skipped 2274999 previous similar messages [Tue Jul 3 08:19:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 08:19:39 2018] Lustre: Skipped 2274999 previous similar messages [Tue Jul 3 08:29:39 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 08:29:39 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 2269238 previous similar messages [Tue Jul 3 08:29:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 08:29:39 2018] Lustre: Skipped 2269239 previous similar messages [Tue Jul 3 08:29:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 08:29:39 2018] Lustre: Skipped 2269239 previous similar messages [Tue Jul 3 08:39:39 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 08:39:39 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 2269893 previous similar messages [Tue Jul 3 08:39:40 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 08:39:40 2018] Lustre: Skipped 2269889 previous similar messages [Tue Jul 3 08:39:40 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 08:39:40 2018] Lustre: Skipped 2269889 previous similar messages [Tue Jul 3 08:49:39 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 08:49:39 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 2277541 previous similar messages [Tue Jul 3 08:49:40 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 08:49:40 2018] Lustre: Skipped 2277544 previous similar messages [Tue Jul 3 08:49:40 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 08:49:40 2018] Lustre: Skipped 2277544 previous similar messages [Tue Jul 3 09:01:00 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 09:01:00 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 2225775 previous similar messages [Tue Jul 3 09:01:00 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 09:01:00 2018] Lustre: Skipped 2225704 previous similar messages [Tue Jul 3 09:01:00 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 09:01:00 2018] Lustre: Skipped 2225704 previous similar messages [Tue Jul 3 09:11:00 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 09:11:00 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 1387742 previous similar messages [Tue Jul 3 09:11:00 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 09:11:00 2018] Lustre: Skipped 1387743 previous similar messages [Tue Jul 3 09:11:00 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 09:11:00 2018] Lustre: Skipped 1387743 previous similar messages [Tue Jul 3 09:21:01 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 09:21:01 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 1826703 previous similar messages [Tue Jul 3 09:21:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 09:21:01 2018] Lustre: Skipped 1826703 previous similar messages [Tue Jul 3 09:21:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 09:21:01 2018] Lustre: Skipped 1826703 previous similar messages [Tue Jul 3 09:31:01 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 09:31:01 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 1835459 previous similar messages [Tue Jul 3 09:31:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 09:31:01 2018] Lustre: Skipped 1835458 previous similar messages [Tue Jul 3 09:31:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 09:31:01 2018] Lustre: Skipped 1835458 previous similar messages [Tue Jul 3 09:35:11 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880b22296000 x1604871149161216/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:555/0 lens 568/440 e 0 to 0 dl 1530574305 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 09:35:11 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffd2fc050 x1604871149161216/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:555/0 lens 568/440 e 0 to 0 dl 1530574305 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 09:35:11 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 3 previous similar messages [Tue Jul 3 09:35:11 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 09:35:11 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 1 previous similar message [Tue Jul 3 09:35:11 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1301 previous similar messages [Tue Jul 3 09:35:14 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880d7fc2ce00 x1604871149499856/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:558/0 lens 568/440 e 0 to 0 dl 1530574308 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 09:35:14 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Tue Jul 3 09:35:21 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8802a0626900 x1604871150298512/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:602/0 lens 568/440 e 0 to 0 dl 1530574352 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 09:35:21 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Tue Jul 3 09:35:32 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2fbc50 x1604871151549376/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:613/0 lens 568/440 e 0 to 0 dl 1530574363 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 09:35:32 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8802d93b8c00 x1604871151549376/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:613/0 lens 568/440 e 0 to 0 dl 1530574363 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 09:35:32 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 11 previous similar messages [Tue Jul 3 09:35:32 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 09:35:32 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 5 previous similar messages [Tue Jul 3 09:35:32 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Tue Jul 3 09:35:56 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe2f6c50 x1604871154363280/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:637/0 lens 568/440 e 0 to 0 dl 1530574387 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 09:35:56 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 4 previous similar messages [Tue Jul 3 09:36:10 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8804d0d96900 x1604871155844000/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:650/0 lens 568/440 e 0 to 0 dl 1530574400 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 09:36:10 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 12 previous similar messages [Tue Jul 3 09:36:12 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 09:36:12 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 6 previous similar messages [Tue Jul 3 09:36:34 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2fc450 x1604871158706336/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:675/0 lens 568/440 e 0 to 0 dl 1530574425 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 09:36:34 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 12 previous similar messages [Tue Jul 3 09:37:48 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffe315c50 x1604871167181728/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:750/0 lens 568/440 e 0 to 0 dl 1530574500 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 09:37:48 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 09:37:48 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 11 previous similar messages [Tue Jul 3 09:37:48 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 25 previous similar messages [Tue Jul 3 09:37:54 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ab00c8c00 x1604871167865408/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:1/0 lens 568/440 e 0 to 0 dl 1530574506 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 09:37:54 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 3 previous similar messages [Tue Jul 3 09:40:20 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880280fada00 x1604871184010688/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:146/0 lens 568/440 e 0 to 0 dl 1530574651 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 09:40:20 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 09:40:20 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 22 previous similar messages [Tue Jul 3 09:40:20 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 48 previous similar messages [Tue Jul 3 09:40:25 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8807b52c3300 x1604871184551856/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:151/0 lens 568/440 e 0 to 0 dl 1530574656 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 09:40:25 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 14 previous similar messages [Tue Jul 3 09:41:01 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 09:41:01 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 6517387 previous similar messages [Tue Jul 3 09:41:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 09:41:01 2018] Lustre: Skipped 6517091 previous similar messages [Tue Jul 3 09:41:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 09:41:01 2018] Lustre: Skipped 6515946 previous similar messages [Tue Jul 3 09:51:01 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 09:51:01 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 5807044 previous similar messages [Tue Jul 3 09:51:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 09:51:01 2018] Lustre: Skipped 5806849 previous similar messages [Tue Jul 3 09:51:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 09:51:01 2018] Lustre: Skipped 5806015 previous similar messages [Tue Jul 3 09:52:51 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff88030aff6900 x1604871257657440/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:105/0 lens 568/440 e 0 to 0 dl 1530575365 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 09:52:51 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffe313450 x1604871257657440/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:105/0 lens 568/440 e 0 to 0 dl 1530575365 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 09:52:51 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 31 previous similar messages [Tue Jul 3 09:52:51 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 09:52:51 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 15 previous similar messages [Tue Jul 3 09:52:51 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 11 previous similar messages [Tue Jul 3 10:01:01 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 10:01:01 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 7057557 previous similar messages [Tue Jul 3 10:01:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 10:01:01 2018] Lustre: Skipped 7057145 previous similar messages [Tue Jul 3 10:01:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 10:01:01 2018] Lustre: Skipped 7055521 previous similar messages [Tue Jul 3 10:11:01 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 10:11:01 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 6849077 previous similar messages [Tue Jul 3 10:11:01 2018] Lustre: lustre-MDT0000: Client 6715f412-2ecd-d612-feba-1973dbec4c82 (at 172.16.229.39@o2ib) reconnecting [Tue Jul 3 10:11:01 2018] Lustre: Skipped 6848663 previous similar messages [Tue Jul 3 10:11:01 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Tue Jul 3 10:11:01 2018] Lustre: Skipped 6847284 previous similar messages [Tue Jul 3 10:11:39 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2fac50 x1604871386384992/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:515/0 lens 568/440 e 0 to 0 dl 1530576530 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 10:11:39 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880f5d9c5d00 x1604871386384992/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:515/0 lens 568/440 e 0 to 0 dl 1530576530 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 10:11:39 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 64 previous similar messages [Tue Jul 3 10:11:39 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 10:11:39 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 27 previous similar messages [Tue Jul 3 10:11:39 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 32 previous similar messages [Tue Jul 3 10:21:01 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 10:21:01 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 5941024 previous similar messages [Tue Jul 3 10:21:01 2018] Lustre: lustre-MDT0000: Client 6715f412-2ecd-d612-feba-1973dbec4c82 (at 172.16.229.39@o2ib) reconnecting [Tue Jul 3 10:21:01 2018] Lustre: Skipped 5940788 previous similar messages [Tue Jul 3 10:21:01 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Tue Jul 3 10:21:01 2018] Lustre: Skipped 5939875 previous similar messages [Tue Jul 3 10:21:41 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880d75828c00 x1604871449878560/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:362/0 lens 568/440 e 0 to 0 dl 1530577132 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 10:21:41 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880319f13300 x1604871449878560/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:362/0 lens 568/440 e 0 to 0 dl 1530577132 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 10:21:41 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 68 previous similar messages [Tue Jul 3 10:21:41 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 10:21:41 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 31 previous similar messages [Tue Jul 3 10:21:41 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 30 previous similar messages [Tue Jul 3 10:31:01 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 10:31:01 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 6687887 previous similar messages [Tue Jul 3 10:31:01 2018] Lustre: lustre-MDT0000: Client 6715f412-2ecd-d612-feba-1973dbec4c82 (at 172.16.229.39@o2ib) reconnecting [Tue Jul 3 10:31:01 2018] Lustre: Skipped 6687546 previous similar messages [Tue Jul 3 10:31:01 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Tue Jul 3 10:31:01 2018] Lustre: Skipped 6686267 previous similar messages [Tue Jul 3 10:32:11 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880cdde70c00 x1604871522101824/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:237/0 lens 568/440 e 0 to 0 dl 1530577762 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 10:32:11 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 10:32:11 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 24 previous similar messages [Tue Jul 3 10:32:11 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 59 previous similar messages [Tue Jul 3 10:32:14 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880b00b80900 x1604871522445296/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:203/0 lens 568/440 e 0 to 0 dl 1530577728 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 10:32:14 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 143 previous similar messages [Tue Jul 3 10:41:01 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 10:41:01 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 5714921 previous similar messages [Tue Jul 3 10:41:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 10:41:01 2018] Lustre: Skipped 5714705 previous similar messages [Tue Jul 3 10:41:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 10:41:01 2018] Lustre: Skipped 5714020 previous similar messages [Tue Jul 3 10:42:11 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ae0a73c00 x1604871591643696/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:81/0 lens 568/440 e 0 to 0 dl 1530578361 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 10:42:11 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 77 previous similar messages [Tue Jul 3 10:42:49 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880847ab2d00 x1604871596074272/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:120/0 lens 568/440 e 0 to 0 dl 1530578400 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 10:42:49 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 10:42:49 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 35 previous similar messages [Tue Jul 3 10:42:49 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 37 previous similar messages [Tue Jul 3 10:51:01 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 10:51:01 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 7027435 previous similar messages [Tue Jul 3 10:51:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 10:51:01 2018] Lustre: Skipped 7027058 previous similar messages [Tue Jul 3 10:51:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 10:51:01 2018] Lustre: Skipped 7025961 previous similar messages [Tue Jul 3 11:01:01 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 11:01:01 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 7055898 previous similar messages [Tue Jul 3 11:01:01 2018] Lustre: lustre-MDT0000: Client 6715f412-2ecd-d612-feba-1973dbec4c82 (at 172.16.229.39@o2ib) reconnecting [Tue Jul 3 11:01:01 2018] Lustre: Skipped 7055562 previous similar messages [Tue Jul 3 11:01:01 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Tue Jul 3 11:01:01 2018] Lustre: Skipped 7054474 previous similar messages [Tue Jul 3 11:06:51 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8802803a6c00 x1604871760429056/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:15/0 lens 568/440 e 0 to 0 dl 1530579805 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 11:06:51 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88022b730f00 x1604871760429056/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:15/0 lens 568/440 e 0 to 0 dl 1530579805 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 11:06:51 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 59 previous similar messages [Tue Jul 3 11:06:51 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 11:06:51 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 24 previous similar messages [Tue Jul 3 11:06:51 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 27 previous similar messages [Tue Jul 3 11:11:01 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 11:11:01 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 7002976 previous similar messages [Tue Jul 3 11:11:01 2018] Lustre: lustre-MDT0000: Client 6715f412-2ecd-d612-feba-1973dbec4c82 (at 172.16.229.39@o2ib) reconnecting [Tue Jul 3 11:11:01 2018] Lustre: Skipped 7002649 previous similar messages [Tue Jul 3 11:11:01 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Tue Jul 3 11:11:01 2018] Lustre: Skipped 7001548 previous similar messages [Tue Jul 3 11:20:27 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880b22290f00 x1604871841671328/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:76/0 lens 568/440 e 0 to 0 dl 1530580621 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 11:20:27 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880b22291200 x1604871841671328/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:76/0 lens 568/440 e 0 to 0 dl 1530580621 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 11:20:27 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 19 previous similar messages [Tue Jul 3 11:20:27 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 11:20:27 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 9 previous similar messages [Tue Jul 3 11:20:27 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 8 previous similar messages [Tue Jul 3 11:20:37 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff88074727a100 x1604871842815728/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:123/0 lens 568/440 e 0 to 0 dl 1530580668 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 11:20:37 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffe316450 x1604871842815728/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:123/0 lens 568/440 e 0 to 0 dl 1530580668 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 11:20:37 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 7 previous similar messages [Tue Jul 3 11:20:37 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 11:20:37 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 3 previous similar messages [Tue Jul 3 11:20:37 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 3 previous similar messages [Tue Jul 3 11:20:56 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8807151de000 x1604871844893344/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:141/0 lens 568/440 e 0 to 0 dl 1530580686 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 11:20:56 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 6 previous similar messages [Tue Jul 3 11:21:01 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 11:21:01 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 5858647 previous similar messages [Tue Jul 3 11:21:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 11:21:01 2018] Lustre: Skipped 5858387 previous similar messages [Tue Jul 3 11:21:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 11:21:01 2018] Lustre: Skipped 5857441 previous similar messages [Tue Jul 3 11:21:03 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880b6c38bf00 x1604871845806672/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:149/0 lens 568/440 e 0 to 0 dl 1530580694 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 11:21:03 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 11:21:03 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 3 previous similar messages [Tue Jul 3 11:21:03 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 4 previous similar messages [Tue Jul 3 11:21:35 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880b00b82700 x1604871849510656/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:181/0 lens 568/440 e 0 to 0 dl 1530580726 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 11:21:35 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 15 previous similar messages [Tue Jul 3 11:21:41 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880868bf1500 x1604871850196144/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:187/0 lens 568/440 e 0 to 0 dl 1530580732 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 11:21:41 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 11:21:41 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 7 previous similar messages [Tue Jul 3 11:21:41 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 3 previous similar messages [Tue Jul 3 11:31:01 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 11:31:01 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 7055663 previous similar messages [Tue Jul 3 11:31:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 11:31:01 2018] Lustre: Skipped 7055322 previous similar messages [Tue Jul 3 11:31:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 11:31:01 2018] Lustre: Skipped 7054061 previous similar messages [Tue Jul 3 11:37:51 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880907ae9800 x1604871960105264/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:402/0 lens 568/440 e 0 to 0 dl 1530581702 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 11:37:51 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88035ca16f00 x1604871960105264/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:402/0 lens 568/440 e 0 to 0 dl 1530581702 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 11:37:51 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 9 previous similar messages [Tue Jul 3 11:37:51 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 11:37:51 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 3 previous similar messages [Tue Jul 3 11:37:51 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 3 previous similar messages [Tue Jul 3 11:38:01 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2f9450 x1604871961237488/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:412/0 lens 568/440 e 0 to 0 dl 1530581712 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 11:38:01 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88102ef66c00 x1604871961237488/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:412/0 lens 568/440 e 0 to 0 dl 1530581712 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 11:38:01 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 7 previous similar messages [Tue Jul 3 11:38:01 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 11:38:01 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 3 previous similar messages [Tue Jul 3 11:38:01 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 3 previous similar messages [Tue Jul 3 11:38:27 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe315c50 x1604871964220144/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:438/0 lens 568/440 e 0 to 0 dl 1530581738 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 11:38:27 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880addb1a700 x1604871964220144/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:438/0 lens 568/440 e 0 to 0 dl 1530581738 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 11:38:27 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 7 previous similar messages [Tue Jul 3 11:38:27 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 11:38:27 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 2 previous similar messages [Tue Jul 3 11:38:27 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 3 previous similar messages [Tue Jul 3 11:39:08 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880b6c38a700 x1604871968741536/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:479/0 lens 568/440 e 0 to 0 dl 1530581779 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 11:39:08 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffd2fd050 x1604871968741536/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:479/0 lens 568/440 e 0 to 0 dl 1530581779 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 11:39:08 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 3 previous similar messages [Tue Jul 3 11:39:08 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 2 previous similar messages [Tue Jul 3 11:39:09 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 11:39:09 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 1 previous similar message [Tue Jul 3 11:41:01 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 11:41:01 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 6978801 previous similar messages [Tue Jul 3 11:41:01 2018] Lustre: lustre-MDT0000: Client 6715f412-2ecd-d612-feba-1973dbec4c82 (at 172.16.229.39@o2ib) reconnecting [Tue Jul 3 11:41:01 2018] Lustre: Skipped 6978488 previous similar messages [Tue Jul 3 11:41:01 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Tue Jul 3 11:41:01 2018] Lustre: Skipped 6977284 previous similar messages [Tue Jul 3 11:47:51 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff88046e6cad00 x1604872027624416/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:247/0 lens 568/440 e 0 to 0 dl 1530582302 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 11:47:51 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880283401800 x1604872027624416/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:247/0 lens 568/440 e 0 to 0 dl 1530582302 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 11:47:51 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 34 previous similar messages [Tue Jul 3 11:47:51 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 18 previous similar messages [Tue Jul 3 11:48:25 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 11:48:25 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 16 previous similar messages [Tue Jul 3 11:51:01 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 11:51:01 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 6520847 previous similar messages [Tue Jul 3 11:51:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 11:51:01 2018] Lustre: Skipped 6520579 previous similar messages [Tue Jul 3 11:51:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 11:51:01 2018] Lustre: Skipped 6519700 previous similar messages [Tue Jul 3 12:01:01 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 12:01:01 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 7004563 previous similar messages [Tue Jul 3 12:01:01 2018] Lustre: lustre-MDT0000: Client 6715f412-2ecd-d612-feba-1973dbec4c82 (at 172.16.229.39@o2ib) reconnecting [Tue Jul 3 12:01:01 2018] Lustre: Skipped 7004197 previous similar messages [Tue Jul 3 12:01:01 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Tue Jul 3 12:01:01 2018] Lustre: Skipped 7002618 previous similar messages [Tue Jul 3 12:07:22 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe316c50 x1604872160257424/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:626/0 lens 568/440 e 0 to 0 dl 1530583436 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 12:07:22 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 326 previous similar messages [Tue Jul 3 12:07:22 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8805c0d39e00 x1604872160257424/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:626/0 lens 568/440 e 0 to 0 dl 1530583436 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 12:07:22 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 31 previous similar messages [Tue Jul 3 12:07:44 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880a252f2a00 x1604872161485008/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:648/0 lens 568/440 e 0 to 0 dl 1530583458 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 12:07:44 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 3300 previous similar messages [Tue Jul 3 12:08:09 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880f527b3f00 x1604872161485008/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:672/0 lens 568/440 e 0 to 0 dl 1530583482 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 12:08:09 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 4 previous similar messages [Tue Jul 3 12:08:26 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880a9f6a3900 x1604872167565904/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:690/0 lens 568/440 e 0 to 0 dl 1530583500 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 12:08:26 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 4750 previous similar messages [Tue Jul 3 12:09:27 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88098f349e00 x1604872168966080/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:751/0 lens 568/440 e 0 to 0 dl 1530583561 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 12:09:27 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 3 previous similar messages [Tue Jul 3 12:09:41 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2f9850 x1604872175845936/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:10/0 lens 568/440 e 0 to 0 dl 1530583575 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 12:09:41 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1639 previous similar messages [Tue Jul 3 12:10:24 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 12:10:24 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 11 previous similar messages [Tue Jul 3 12:11:01 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 12:11:01 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 6521826 previous similar messages [Tue Jul 3 12:11:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 12:11:01 2018] Lustre: Skipped 6521517 previous similar messages [Tue Jul 3 12:11:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 12:11:01 2018] Lustre: Skipped 6520414 previous similar messages [Tue Jul 3 12:11:28 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8806cd2e7800 x1604872183144528/t0(0) o37->6715f412-2ecd-d612-feba-1973dbec4c82@172.16.229.39@o2ib:154/0 lens 568/440 e 0 to 0 dl 1530583719 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 12:11:28 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 12 previous similar messages [Tue Jul 3 12:11:28 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Tue Jul 3 12:12:31 2018] LNet: 25131:0:(o2iblnd_cb.c:2502:kiblnd_passive_connect()) Conn stale 172.16.229.39@o2ib version 12/12 incarnation 1530524375553037/1530583738544115 [Tue Jul 3 12:13:13 2018] LustreError: 2852:0:(ldlm_lockd.c:331:waiting_locks_callback()) ### lock callback timer expired after 149s: evicting client at 172.16.229.39@o2ib ns: mdt-lustre-MDT0000_UUID lock: ffff880ed9b85100/0x88fac5b5c265eb7 lrc: 3/0,0 mode: PR/PR res: [0x2000131a3:0x17866:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT flags: 0x60000400000020 nid: 172.16.229.39@o2ib remote: 0x372ce96e19d9cec0 expref: 3072 pid: 89767 timeout: 3006179 lvb_type: 0 [Tue Jul 3 12:15:02 2018] Lustre: MGS: haven't heard from client b03e06a9-4e24-f603-0c2e-aff612aac478 (at 172.16.229.39@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881a50698400, cur 1530583890 expire 1530583740 last 1530583663 [Tue Jul 3 12:22:09 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 12:22:09 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 2394817 previous similar messages [Tue Jul 3 12:22:09 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 12:22:09 2018] Lustre: Skipped 2394798 previous similar messages [Tue Jul 3 12:22:09 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 12:22:09 2018] Lustre: Skipped 2394689 previous similar messages [Tue Jul 3 12:32:09 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 12:32:09 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 2287768 previous similar messages [Tue Jul 3 12:32:09 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 12:32:09 2018] Lustre: Skipped 2287768 previous similar messages [Tue Jul 3 12:32:09 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 12:32:09 2018] Lustre: Skipped 2287768 previous similar messages [Tue Jul 3 12:42:09 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 12:42:09 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2278283 previous similar messages [Tue Jul 3 12:42:09 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 12:42:09 2018] Lustre: Skipped 2278283 previous similar messages [Tue Jul 3 12:42:09 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 12:42:09 2018] Lustre: Skipped 2278283 previous similar messages [Tue Jul 3 12:44:14 2018] usb 1-1.6.1: new high-speed USB device number 5 using ehci-pci [Tue Jul 3 12:44:14 2018] usb 1-1.6.1: New USB device found, idVendor=0624, idProduct=0249 [Tue Jul 3 12:44:14 2018] usb 1-1.6.1: New USB device strings: Mfr=4, Product=5, SerialNumber=6 [Tue Jul 3 12:44:14 2018] usb 1-1.6.1: Product: Keyboard/Mouse Function [Tue Jul 3 12:44:14 2018] usb 1-1.6.1: Manufacturer: Avocent [Tue Jul 3 12:44:14 2018] usb 1-1.6.1: SerialNumber: 20121018 [Tue Jul 3 12:44:14 2018] input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1-1.6.1:1.0/input/input2 [Tue Jul 3 12:44:14 2018] hid-generic 0003:0624:0249.0001: input,hidraw0: USB HID v1.00 Keyboard [Avocent Keyboard/Mouse Function] on usb-0000:00:1a.0-1.6.1/input0 [Tue Jul 3 12:44:14 2018] input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1-1.6.1:1.1/input/input3 [Tue Jul 3 12:44:14 2018] hid-generic 0003:0624:0249.0002: input,hidraw1: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] on usb-0000:00:1a.0-1.6.1/input1 [Tue Jul 3 12:44:14 2018] input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1-1.6.1:1.2/input/input4 [Tue Jul 3 12:44:14 2018] hid-generic 0003:0624:0249.0003: input,hidraw2: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] on usb-0000:00:1a.0-1.6.1/input2 [Tue Jul 3 12:47:09 2018] igb 0000:01:00.0: changing MTU from 9000 to 1500 [Tue Jul 3 12:47:09 2018] IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready [Tue Jul 3 12:47:14 2018] igb 0000:01:00.0 em1: igb: em1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [Tue Jul 3 12:47:14 2018] IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready [Tue Jul 3 12:47:15 2018] igb 0000:01:00.0: changing MTU from 1500 to 9000 [Tue Jul 3 12:47:19 2018] igb 0000:01:00.0 em1: igb: em1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [Tue Jul 3 12:48:40 2018] usb 1-1.6.1: USB disconnect, device number 5 [Tue Jul 3 12:52:09 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 12:52:09 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 2288340 previous similar messages [Tue Jul 3 12:52:09 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 12:52:09 2018] Lustre: Skipped 2288349 previous similar messages [Tue Jul 3 12:52:09 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 12:52:09 2018] Lustre: Skipped 2288349 previous similar messages [Tue Jul 3 13:02:09 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 13:02:09 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 2294669 previous similar messages [Tue Jul 3 13:02:09 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 13:02:09 2018] Lustre: Skipped 2294668 previous similar messages [Tue Jul 3 13:02:09 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 13:02:09 2018] Lustre: Skipped 2294668 previous similar messages [Tue Jul 3 13:12:09 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 13:12:09 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 2313644 previous similar messages [Tue Jul 3 13:12:09 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 13:12:09 2018] Lustre: Skipped 2313644 previous similar messages [Tue Jul 3 13:12:09 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 13:12:09 2018] Lustre: Skipped 2313644 previous similar messages [Tue Jul 3 13:22:09 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 13:22:09 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 2262392 previous similar messages [Tue Jul 3 13:22:09 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 13:22:09 2018] Lustre: Skipped 2262392 previous similar messages [Tue Jul 3 13:22:09 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 13:22:09 2018] Lustre: Skipped 2262392 previous similar messages [Tue Jul 3 13:32:09 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 13:32:09 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 2291365 previous similar messages [Tue Jul 3 13:32:09 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 13:32:09 2018] Lustre: Skipped 2291364 previous similar messages [Tue Jul 3 13:32:09 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 13:32:09 2018] Lustre: Skipped 2291364 previous similar messages [Tue Jul 3 13:42:09 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 13:42:09 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 2256771 previous similar messages [Tue Jul 3 13:42:09 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 13:42:09 2018] Lustre: Skipped 2256773 previous similar messages [Tue Jul 3 13:42:09 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 13:42:09 2018] Lustre: Skipped 2256773 previous similar messages [Tue Jul 3 13:52:58 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 13:52:58 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 2075682 previous similar messages [Tue Jul 3 13:52:58 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 13:52:58 2018] Lustre: Skipped 2075673 previous similar messages [Tue Jul 3 13:52:58 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 13:52:58 2018] Lustre: Skipped 2075673 previous similar messages [Tue Jul 3 14:03:07 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 14:03:07 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 487698 previous similar messages [Tue Jul 3 14:03:07 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 14:03:07 2018] Lustre: Skipped 487698 previous similar messages [Tue Jul 3 14:03:07 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 14:03:07 2018] Lustre: Skipped 487698 previous similar messages [Tue Jul 3 14:13:27 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 14:13:27 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 1461570 previous similar messages [Tue Jul 3 14:13:28 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 14:13:28 2018] Lustre: Skipped 1461570 previous similar messages [Tue Jul 3 14:13:28 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 14:13:28 2018] Lustre: Skipped 1461570 previous similar messages [Tue Jul 3 14:24:07 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 14:24:07 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 1084085 previous similar messages [Tue Jul 3 14:24:07 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 14:24:07 2018] Lustre: Skipped 1084085 previous similar messages [Tue Jul 3 14:24:07 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 14:24:07 2018] Lustre: Skipped 1084085 previous similar messages [Tue Jul 3 14:34:07 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 14:34:07 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 2351459 previous similar messages [Tue Jul 3 14:34:07 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 14:34:07 2018] Lustre: Skipped 2351459 previous similar messages [Tue Jul 3 14:34:07 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 14:34:07 2018] Lustre: Skipped 2351459 previous similar messages [Tue Jul 3 14:44:07 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 14:44:07 2018] Lustre: Skipped 2280720 previous similar messages [Tue Jul 3 14:44:07 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 14:44:07 2018] Lustre: Skipped 2280720 previous similar messages [Tue Jul 3 14:44:07 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 14:44:07 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2280721 previous similar messages [Tue Jul 3 14:55:31 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 14:55:31 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 1312328 previous similar messages [Tue Jul 3 14:55:31 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 14:55:31 2018] Lustre: Skipped 1312329 previous similar messages [Tue Jul 3 14:55:31 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 14:55:31 2018] Lustre: Skipped 1312329 previous similar messages [Tue Jul 3 15:05:58 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 15:05:58 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 629363 previous similar messages [Tue Jul 3 15:05:58 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 15:05:58 2018] Lustre: Skipped 629363 previous similar messages [Tue Jul 3 15:05:58 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 15:05:58 2018] Lustre: Skipped 629363 previous similar messages [Tue Jul 3 15:16:01 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 15:16:01 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 899455 previous similar messages [Tue Jul 3 15:16:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 15:16:01 2018] Lustre: Skipped 899455 previous similar messages [Tue Jul 3 15:16:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 15:16:01 2018] Lustre: Skipped 899455 previous similar messages [Tue Jul 3 15:26:02 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 15:26:02 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 1380856 previous similar messages [Tue Jul 3 15:26:02 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 15:26:02 2018] Lustre: Skipped 1380856 previous similar messages [Tue Jul 3 15:26:02 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 15:26:02 2018] Lustre: Skipped 1380856 previous similar messages [Tue Jul 3 15:36:03 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 15:36:03 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 1852266 previous similar messages [Tue Jul 3 15:36:03 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 15:36:03 2018] Lustre: Skipped 1852266 previous similar messages [Tue Jul 3 15:36:03 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 15:36:03 2018] Lustre: Skipped 1852266 previous similar messages [Tue Jul 3 15:46:03 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 15:46:03 2018] Lustre: Skipped 1341036 previous similar messages [Tue Jul 3 15:46:03 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 15:46:03 2018] Lustre: Skipped 1341036 previous similar messages [Tue Jul 3 15:46:03 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 15:46:03 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 1341037 previous similar messages [Tue Jul 3 15:57:30 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 15:57:30 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 1322278 previous similar messages [Tue Jul 3 15:57:30 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 15:57:30 2018] Lustre: Skipped 1322279 previous similar messages [Tue Jul 3 15:57:30 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 15:57:30 2018] Lustre: Skipped 1322279 previous similar messages [Tue Jul 3 16:08:03 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 16:08:03 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 1086618 previous similar messages [Tue Jul 3 16:08:03 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 16:08:03 2018] Lustre: Skipped 1086618 previous similar messages [Tue Jul 3 16:08:03 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 16:08:03 2018] Lustre: Skipped 1086618 previous similar messages [Tue Jul 3 16:18:03 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 16:18:03 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 444952 previous similar messages [Tue Jul 3 16:18:03 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 16:18:03 2018] Lustre: Skipped 444952 previous similar messages [Tue Jul 3 16:18:03 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 16:18:03 2018] Lustre: Skipped 444952 previous similar messages [Tue Jul 3 16:28:03 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 16:28:03 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 2288240 previous similar messages [Tue Jul 3 16:28:03 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 16:28:03 2018] Lustre: Skipped 2288240 previous similar messages [Tue Jul 3 16:28:03 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 16:28:03 2018] Lustre: Skipped 2288240 previous similar messages [Tue Jul 3 16:38:03 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 16:38:03 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 2309862 previous similar messages [Tue Jul 3 16:38:03 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 16:38:03 2018] Lustre: Skipped 2309862 previous similar messages [Tue Jul 3 16:38:03 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 16:38:03 2018] Lustre: Skipped 2309862 previous similar messages [Tue Jul 3 16:48:03 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 16:48:03 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 1861680 previous similar messages [Tue Jul 3 16:48:03 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 16:48:03 2018] Lustre: Skipped 1861871 previous similar messages [Tue Jul 3 16:48:03 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 16:48:03 2018] Lustre: Skipped 1861871 previous similar messages [Tue Jul 3 16:58:47 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 16:58:47 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 1631391 previous similar messages [Tue Jul 3 16:58:47 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 16:58:47 2018] Lustre: Skipped 1631200 previous similar messages [Tue Jul 3 16:58:47 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 16:58:47 2018] Lustre: Skipped 1631200 previous similar messages [Tue Jul 3 17:10:26 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 17:10:26 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 1814782 previous similar messages [Tue Jul 3 17:10:26 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 17:10:26 2018] Lustre: Skipped 1814782 previous similar messages [Tue Jul 3 17:10:26 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 17:10:26 2018] Lustre: Skipped 1814782 previous similar messages [Tue Jul 3 17:21:43 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 17:21:43 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 1279322 previous similar messages [Tue Jul 3 17:21:44 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 17:21:44 2018] Lustre: Skipped 1279322 previous similar messages [Tue Jul 3 17:21:44 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 17:21:44 2018] Lustre: Skipped 1279322 previous similar messages [Tue Jul 3 17:31:44 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 17:31:44 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 2272767 previous similar messages [Tue Jul 3 17:31:44 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 17:31:44 2018] Lustre: Skipped 2272769 previous similar messages [Tue Jul 3 17:31:44 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 17:31:44 2018] Lustre: Skipped 2272769 previous similar messages [Tue Jul 3 17:41:44 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 17:41:44 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 1811744 previous similar messages [Tue Jul 3 17:41:44 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 17:41:44 2018] Lustre: Skipped 1811745 previous similar messages [Tue Jul 3 17:41:44 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 17:41:44 2018] Lustre: Skipped 1811745 previous similar messages [Tue Jul 3 17:52:06 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 17:52:06 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 1464949 previous similar messages [Tue Jul 3 17:52:06 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 17:52:06 2018] Lustre: Skipped 1464946 previous similar messages [Tue Jul 3 17:52:06 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 17:52:06 2018] Lustre: Skipped 1464946 previous similar messages [Tue Jul 3 18:02:06 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 18:02:06 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 2373515 previous similar messages [Tue Jul 3 18:02:06 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 18:02:06 2018] Lustre: Skipped 2373515 previous similar messages [Tue Jul 3 18:02:06 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 18:02:06 2018] Lustre: Skipped 2373515 previous similar messages [Tue Jul 3 18:12:06 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 18:12:06 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 462589 previous similar messages [Tue Jul 3 18:12:06 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 18:12:06 2018] Lustre: Skipped 462589 previous similar messages [Tue Jul 3 18:12:06 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 18:12:06 2018] Lustre: Skipped 462589 previous similar messages [Tue Jul 3 18:22:06 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 18:22:06 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 1844802 previous similar messages [Tue Jul 3 18:22:06 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 18:22:06 2018] Lustre: Skipped 1844802 previous similar messages [Tue Jul 3 18:22:06 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 18:22:06 2018] Lustre: Skipped 1844802 previous similar messages [Tue Jul 3 18:32:06 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 18:32:06 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 2308064 previous similar messages [Tue Jul 3 18:32:06 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 18:32:06 2018] Lustre: Skipped 2308067 previous similar messages [Tue Jul 3 18:32:06 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 18:32:06 2018] Lustre: Skipped 2308067 previous similar messages [Tue Jul 3 18:42:06 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 18:42:06 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 2285834 previous similar messages [Tue Jul 3 18:42:06 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 18:42:06 2018] Lustre: Skipped 2285833 previous similar messages [Tue Jul 3 18:42:06 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 18:42:06 2018] Lustre: Skipped 2285833 previous similar messages [Tue Jul 3 18:52:45 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 18:52:45 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 2034712 previous similar messages [Tue Jul 3 18:52:45 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 18:52:45 2018] Lustre: Skipped 2034710 previous similar messages [Tue Jul 3 18:52:45 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 18:52:45 2018] Lustre: Skipped 2034710 previous similar messages [Tue Jul 3 19:04:05 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 19:04:05 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 2312519 previous similar messages [Tue Jul 3 19:04:05 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 19:04:05 2018] Lustre: Skipped 2312519 previous similar messages [Tue Jul 3 19:04:05 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 19:04:05 2018] Lustre: Skipped 2312519 previous similar messages [Tue Jul 3 19:14:05 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 19:14:05 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 2280074 previous similar messages [Tue Jul 3 19:14:05 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 19:14:05 2018] Lustre: Skipped 2280076 previous similar messages [Tue Jul 3 19:14:05 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 19:14:05 2018] Lustre: Skipped 2280076 previous similar messages [Tue Jul 3 19:24:05 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 19:24:05 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 1815327 previous similar messages [Tue Jul 3 19:24:06 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 19:24:06 2018] Lustre: Skipped 1815326 previous similar messages [Tue Jul 3 19:24:06 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 19:24:06 2018] Lustre: Skipped 1815326 previous similar messages [Tue Jul 3 19:35:26 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 19:35:26 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 1764478 previous similar messages [Tue Jul 3 19:35:26 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 19:35:26 2018] Lustre: Skipped 1764477 previous similar messages [Tue Jul 3 19:35:26 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 19:35:26 2018] Lustre: Skipped 1764477 previous similar messages [Tue Jul 3 19:45:26 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 19:45:26 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 1858410 previous similar messages [Tue Jul 3 19:45:26 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 19:45:26 2018] Lustre: Skipped 1858411 previous similar messages [Tue Jul 3 19:45:26 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 19:45:26 2018] Lustre: Skipped 1858411 previous similar messages [Tue Jul 3 19:55:26 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 19:55:26 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 2259271 previous similar messages [Tue Jul 3 19:55:26 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 19:55:26 2018] Lustre: Skipped 2259344 previous similar messages [Tue Jul 3 19:55:26 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 19:55:26 2018] Lustre: Skipped 2259344 previous similar messages [Tue Jul 3 20:05:26 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 20:05:26 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 1843780 previous similar messages [Tue Jul 3 20:05:26 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 20:05:26 2018] Lustre: Skipped 1843782 previous similar messages [Tue Jul 3 20:05:26 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 20:05:26 2018] Lustre: Skipped 1843782 previous similar messages [Tue Jul 3 20:15:26 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 20:15:26 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 2328378 previous similar messages [Tue Jul 3 20:15:26 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 20:15:26 2018] Lustre: Skipped 2328376 previous similar messages [Tue Jul 3 20:15:26 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 20:15:26 2018] Lustre: Skipped 2328376 previous similar messages [Tue Jul 3 20:25:53 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 20:25:53 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 1031136 previous similar messages [Tue Jul 3 20:25:53 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 20:25:53 2018] Lustre: Skipped 1031062 previous similar messages [Tue Jul 3 20:25:53 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 20:25:53 2018] Lustre: Skipped 1031062 previous similar messages [Tue Jul 3 20:35:53 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 20:35:53 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 1875596 previous similar messages [Tue Jul 3 20:35:53 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 20:35:53 2018] Lustre: Skipped 1875597 previous similar messages [Tue Jul 3 20:35:53 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 20:35:53 2018] Lustre: Skipped 1875597 previous similar messages [Tue Jul 3 20:45:53 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 20:45:53 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 1860598 previous similar messages [Tue Jul 3 20:45:53 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 20:45:53 2018] Lustre: Skipped 1860599 previous similar messages [Tue Jul 3 20:45:53 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 20:45:53 2018] Lustre: Skipped 1860599 previous similar messages [Tue Jul 3 20:55:53 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 20:55:53 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 2310575 previous similar messages [Tue Jul 3 20:55:53 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 20:55:53 2018] Lustre: Skipped 2310733 previous similar messages [Tue Jul 3 20:55:53 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 20:55:53 2018] Lustre: Skipped 2310734 previous similar messages [Tue Jul 3 21:05:53 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 21:05:53 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 1917837 previous similar messages [Tue Jul 3 21:05:53 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 21:05:53 2018] Lustre: Skipped 1917679 previous similar messages [Tue Jul 3 21:05:53 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 21:05:53 2018] Lustre: Skipped 1917680 previous similar messages [Tue Jul 3 21:16:06 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 21:16:06 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 1898186 previous similar messages [Tue Jul 3 21:16:06 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 21:16:06 2018] Lustre: Skipped 1898183 previous similar messages [Tue Jul 3 21:16:06 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 21:16:06 2018] Lustre: Skipped 1898183 previous similar messages [Tue Jul 3 21:27:03 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 21:27:03 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 1646473 previous similar messages [Tue Jul 3 21:27:03 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 21:27:03 2018] Lustre: Skipped 1646473 previous similar messages [Tue Jul 3 21:27:03 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 21:27:03 2018] Lustre: Skipped 1646473 previous similar messages [Tue Jul 3 21:37:23 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 21:37:23 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 1878557 previous similar messages [Tue Jul 3 21:37:23 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 21:37:23 2018] Lustre: Skipped 1878557 previous similar messages [Tue Jul 3 21:37:23 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 21:37:23 2018] Lustre: Skipped 1878557 previous similar messages [Tue Jul 3 21:47:23 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 21:47:23 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 1836957 previous similar messages [Tue Jul 3 21:47:23 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 21:47:23 2018] Lustre: Skipped 1836960 previous similar messages [Tue Jul 3 21:47:23 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 21:47:23 2018] Lustre: Skipped 1836960 previous similar messages [Tue Jul 3 21:58:25 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 21:58:25 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 1218730 previous similar messages [Tue Jul 3 21:58:25 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 21:58:25 2018] Lustre: Skipped 1218727 previous similar messages [Tue Jul 3 21:58:25 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 21:58:25 2018] Lustre: Skipped 1218727 previous similar messages [Tue Jul 3 22:09:56 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 22:09:56 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 2237148 previous similar messages [Tue Jul 3 22:09:56 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 22:09:56 2018] Lustre: Skipped 2237148 previous similar messages [Tue Jul 3 22:09:56 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 22:09:56 2018] Lustre: Skipped 2237148 previous similar messages [Tue Jul 3 22:19:56 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 22:19:56 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 1866495 previous similar messages [Tue Jul 3 22:19:56 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 22:19:56 2018] Lustre: Skipped 1866495 previous similar messages [Tue Jul 3 22:19:56 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 22:19:56 2018] Lustre: Skipped 1866495 previous similar messages [Tue Jul 3 22:29:56 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 22:29:56 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 2301085 previous similar messages [Tue Jul 3 22:29:56 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 22:29:56 2018] Lustre: Skipped 2301089 previous similar messages [Tue Jul 3 22:29:56 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 22:29:56 2018] Lustre: Skipped 2301089 previous similar messages [Tue Jul 3 22:39:56 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 22:39:56 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 1830075 previous similar messages [Tue Jul 3 22:39:56 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 22:39:56 2018] Lustre: Skipped 1830109 previous similar messages [Tue Jul 3 22:39:56 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 22:39:56 2018] Lustre: Skipped 1830109 previous similar messages [Tue Jul 3 22:49:56 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 22:49:56 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 2268075 previous similar messages [Tue Jul 3 22:49:57 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 22:49:57 2018] Lustre: Skipped 2268075 previous similar messages [Tue Jul 3 22:49:57 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 22:49:57 2018] Lustre: Skipped 2268075 previous similar messages [Tue Jul 3 22:59:57 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 22:59:57 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 2449062 previous similar messages [Tue Jul 3 22:59:57 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 22:59:57 2018] Lustre: Skipped 2449059 previous similar messages [Tue Jul 3 22:59:57 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 22:59:57 2018] Lustre: Skipped 2449059 previous similar messages [Tue Jul 3 23:09:57 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 23:09:57 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 1858045 previous similar messages [Tue Jul 3 23:09:57 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 23:09:57 2018] Lustre: Skipped 1858010 previous similar messages [Tue Jul 3 23:09:57 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 23:09:57 2018] Lustre: Skipped 1858010 previous similar messages [Tue Jul 3 23:19:57 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 23:19:57 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 2315406 previous similar messages [Tue Jul 3 23:19:57 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 23:19:57 2018] Lustre: Skipped 2315410 previous similar messages [Tue Jul 3 23:19:57 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 23:19:57 2018] Lustre: Skipped 2315410 previous similar messages [Tue Jul 3 23:20:35 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2fdc50 x1604323232352384/t0(0) o37->74dc27c9-7eb7-6236-850f-6507aace669b@172.16.230.55@o2ib:246/0 lens 568/440 e 0 to 0 dl 1530623826 ref 1 fl Interpret:/0/0 rc 0/0 [Tue Jul 3 23:20:35 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880c4bf2a700 x1604323232352384/t0(0) o37->74dc27c9-7eb7-6236-850f-6507aace669b@172.16.230.55@o2ib:246/0 lens 568/440 e 0 to 0 dl 1530623826 ref 1 fl Interpret:/2/0 rc 0/0 [Tue Jul 3 23:20:35 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 1 previous similar message [Tue Jul 3 23:20:35 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.230.55@o2ib [Tue Jul 3 23:20:35 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 5453 previous similar messages [Tue Jul 3 23:20:41 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.230.55@o2ib [Tue Jul 3 23:29:57 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 23:29:57 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 2305056 previous similar messages [Tue Jul 3 23:29:57 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 23:29:57 2018] Lustre: Skipped 2305054 previous similar messages [Tue Jul 3 23:29:57 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 23:29:57 2018] Lustre: Skipped 2305054 previous similar messages [Tue Jul 3 23:39:57 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 23:39:57 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2282111 previous similar messages [Tue Jul 3 23:39:57 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 23:39:57 2018] Lustre: Skipped 2282111 previous similar messages [Tue Jul 3 23:39:57 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 23:39:57 2018] Lustre: Skipped 2282111 previous similar messages [Tue Jul 3 23:49:58 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 23:49:58 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 2289064 previous similar messages [Tue Jul 3 23:49:58 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 23:49:58 2018] Lustre: Skipped 2289062 previous similar messages [Tue Jul 3 23:49:58 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 23:49:58 2018] Lustre: Skipped 2289062 previous similar messages [Tue Jul 3 23:59:58 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Tue Jul 3 23:59:58 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 2275313 previous similar messages [Tue Jul 3 23:59:58 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Tue Jul 3 23:59:58 2018] Lustre: Skipped 2275314 previous similar messages [Tue Jul 3 23:59:58 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Tue Jul 3 23:59:58 2018] Lustre: Skipped 2275314 previous similar messages [Wed Jul 4 00:09:58 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 00:09:58 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 1822466 previous similar messages [Wed Jul 4 00:09:58 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 00:09:58 2018] Lustre: Skipped 1822466 previous similar messages [Wed Jul 4 00:09:58 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 00:09:58 2018] Lustre: Skipped 1822466 previous similar messages [Wed Jul 4 00:19:58 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 00:19:58 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 2316840 previous similar messages [Wed Jul 4 00:19:58 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 00:19:58 2018] Lustre: Skipped 2316839 previous similar messages [Wed Jul 4 00:19:58 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 00:19:58 2018] Lustre: Skipped 2316839 previous similar messages [Wed Jul 4 00:29:58 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 00:29:58 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 1845478 previous similar messages [Wed Jul 4 00:29:58 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 00:29:58 2018] Lustre: Skipped 1845478 previous similar messages [Wed Jul 4 00:29:58 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 00:29:58 2018] Lustre: Skipped 1845478 previous similar messages [Wed Jul 4 00:39:58 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 00:39:58 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 1860859 previous similar messages [Wed Jul 4 00:39:58 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 00:39:58 2018] Lustre: Skipped 1860876 previous similar messages [Wed Jul 4 00:39:58 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 00:39:58 2018] Lustre: Skipped 1860876 previous similar messages [Wed Jul 4 00:49:59 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 00:49:59 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 2334298 previous similar messages [Wed Jul 4 00:49:59 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 00:49:59 2018] Lustre: Skipped 2334281 previous similar messages [Wed Jul 4 00:49:59 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 00:49:59 2018] Lustre: Skipped 2334281 previous similar messages [Wed Jul 4 00:59:59 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 00:59:59 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 2332068 previous similar messages [Wed Jul 4 00:59:59 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 00:59:59 2018] Lustre: Skipped 2332068 previous similar messages [Wed Jul 4 00:59:59 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 00:59:59 2018] Lustre: Skipped 2332068 previous similar messages [Wed Jul 4 01:09:59 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 01:09:59 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 2295474 previous similar messages [Wed Jul 4 01:09:59 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 01:09:59 2018] Lustre: Skipped 2295474 previous similar messages [Wed Jul 4 01:09:59 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 01:09:59 2018] Lustre: Skipped 2295474 previous similar messages [Wed Jul 4 01:19:59 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 01:19:59 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 1855878 previous similar messages [Wed Jul 4 01:19:59 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 01:19:59 2018] Lustre: Skipped 1855880 previous similar messages [Wed Jul 4 01:19:59 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 01:19:59 2018] Lustre: Skipped 1855885 previous similar messages [Wed Jul 4 01:29:59 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 01:29:59 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 2280197 previous similar messages [Wed Jul 4 01:29:59 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 01:29:59 2018] Lustre: Skipped 2280195 previous similar messages [Wed Jul 4 01:29:59 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 01:29:59 2018] Lustre: Skipped 2280191 previous similar messages [Wed Jul 4 01:39:59 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 01:39:59 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 2309436 previous similar messages [Wed Jul 4 01:39:59 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 01:39:59 2018] Lustre: Skipped 2309470 previous similar messages [Wed Jul 4 01:39:59 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 01:39:59 2018] Lustre: Skipped 2309470 previous similar messages [Wed Jul 4 01:49:59 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 01:49:59 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 2297842 previous similar messages [Wed Jul 4 01:49:59 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 01:49:59 2018] Lustre: Skipped 2297843 previous similar messages [Wed Jul 4 01:49:59 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 01:49:59 2018] Lustre: Skipped 2297845 previous similar messages [Wed Jul 4 01:59:59 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 01:59:59 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 2332556 previous similar messages [Wed Jul 4 01:59:59 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 01:59:59 2018] Lustre: Skipped 2332554 previous similar messages [Wed Jul 4 01:59:59 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 01:59:59 2018] Lustre: Skipped 2332552 previous similar messages [Wed Jul 4 02:10:00 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 02:10:00 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 2291390 previous similar messages [Wed Jul 4 02:10:00 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 02:10:00 2018] Lustre: Skipped 2291357 previous similar messages [Wed Jul 4 02:10:00 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 02:10:00 2018] Lustre: Skipped 2291356 previous similar messages [Wed Jul 4 02:20:00 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 02:20:00 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 2316779 previous similar messages [Wed Jul 4 02:20:00 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 02:20:00 2018] Lustre: Skipped 2316781 previous similar messages [Wed Jul 4 02:20:00 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 02:20:00 2018] Lustre: Skipped 2316781 previous similar messages [Wed Jul 4 02:30:00 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 02:30:00 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 2330442 previous similar messages [Wed Jul 4 02:30:00 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 02:30:00 2018] Lustre: Skipped 2330441 previous similar messages [Wed Jul 4 02:30:00 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 02:30:00 2018] Lustre: Skipped 2330441 previous similar messages [Wed Jul 4 02:40:00 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 02:40:00 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 2331783 previous similar messages [Wed Jul 4 02:40:00 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 02:40:00 2018] Lustre: Skipped 2331783 previous similar messages [Wed Jul 4 02:40:00 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 02:40:00 2018] Lustre: Skipped 2331783 previous similar messages [Wed Jul 4 02:50:00 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 02:50:00 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 2290891 previous similar messages [Wed Jul 4 02:50:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 02:50:01 2018] Lustre: Skipped 2290891 previous similar messages [Wed Jul 4 02:50:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 02:50:01 2018] Lustre: Skipped 2290891 previous similar messages [Wed Jul 4 03:00:01 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 03:00:01 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 2391805 previous similar messages [Wed Jul 4 03:00:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 03:00:01 2018] Lustre: Skipped 2391813 previous similar messages [Wed Jul 4 03:00:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 03:00:01 2018] Lustre: Skipped 2391813 previous similar messages [Wed Jul 4 03:10:01 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 03:10:01 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 2296150 previous similar messages [Wed Jul 4 03:10:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 03:10:01 2018] Lustre: Skipped 2296151 previous similar messages [Wed Jul 4 03:10:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 03:10:01 2018] Lustre: Skipped 2296151 previous similar messages [Wed Jul 4 03:20:01 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 03:20:01 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 2255444 previous similar messages [Wed Jul 4 03:20:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 03:20:01 2018] Lustre: Skipped 2255441 previous similar messages [Wed Jul 4 03:20:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 03:20:01 2018] Lustre: Skipped 2255441 previous similar messages [Wed Jul 4 03:30:01 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 03:30:01 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 2337062 previous similar messages [Wed Jul 4 03:30:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 03:30:01 2018] Lustre: Skipped 2337057 previous similar messages [Wed Jul 4 03:30:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 03:30:01 2018] Lustre: Skipped 2337057 previous similar messages [Wed Jul 4 03:40:01 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 03:40:01 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 2339857 previous similar messages [Wed Jul 4 03:40:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 03:40:01 2018] Lustre: Skipped 2339883 previous similar messages [Wed Jul 4 03:40:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 03:40:01 2018] Lustre: Skipped 2339883 previous similar messages [Wed Jul 4 03:50:01 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 03:50:01 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 1829302 previous similar messages [Wed Jul 4 03:50:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 03:50:01 2018] Lustre: Skipped 1829302 previous similar messages [Wed Jul 4 03:50:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 03:50:01 2018] Lustre: Skipped 1829302 previous similar messages [Wed Jul 4 04:00:01 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 04:00:01 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2379796 previous similar messages [Wed Jul 4 04:00:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 04:00:01 2018] Lustre: Skipped 2379797 previous similar messages [Wed Jul 4 04:00:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 04:00:01 2018] Lustre: Skipped 2379797 previous similar messages [Wed Jul 4 04:10:01 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 04:10:01 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 2323902 previous similar messages [Wed Jul 4 04:10:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 04:10:01 2018] Lustre: Skipped 2323898 previous similar messages [Wed Jul 4 04:10:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 04:10:01 2018] Lustre: Skipped 2323898 previous similar messages [Wed Jul 4 04:20:01 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 04:20:01 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2310347 previous similar messages [Wed Jul 4 04:20:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 04:20:01 2018] Lustre: Skipped 2310351 previous similar messages [Wed Jul 4 04:20:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 04:20:01 2018] Lustre: Skipped 2310351 previous similar messages [Wed Jul 4 04:30:01 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 04:30:01 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 2291309 previous similar messages [Wed Jul 4 04:30:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 04:30:01 2018] Lustre: Skipped 2291309 previous similar messages [Wed Jul 4 04:30:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 04:30:01 2018] Lustre: Skipped 2291309 previous similar messages [Wed Jul 4 04:40:01 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 04:40:01 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 2312094 previous similar messages [Wed Jul 4 04:40:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 04:40:01 2018] Lustre: Skipped 2312094 previous similar messages [Wed Jul 4 04:40:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 04:40:01 2018] Lustre: Skipped 2312094 previous similar messages [Wed Jul 4 04:50:01 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 04:50:01 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 2302495 previous similar messages [Wed Jul 4 04:50:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 04:50:01 2018] Lustre: Skipped 2302490 previous similar messages [Wed Jul 4 04:50:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 04:50:01 2018] Lustre: Skipped 2302490 previous similar messages [Wed Jul 4 05:00:01 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 05:00:01 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 1856976 previous similar messages [Wed Jul 4 05:00:01 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 05:00:01 2018] Lustre: Skipped 1856982 previous similar messages [Wed Jul 4 05:00:01 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 05:00:01 2018] Lustre: Skipped 1856982 previous similar messages [Wed Jul 4 05:10:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 05:10:37 2018] Lustre: Skipped 2255031 previous similar messages [Wed Jul 4 05:10:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 05:10:37 2018] Lustre: Skipped 2255031 previous similar messages [Wed Jul 4 05:10:37 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 05:10:37 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 2255062 previous similar messages [Wed Jul 4 05:20:37 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 05:20:37 2018] Lustre: Skipped 2294734 previous similar messages [Wed Jul 4 05:20:37 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 05:20:37 2018] Lustre: Skipped 2294734 previous similar messages [Wed Jul 4 05:20:38 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 05:20:38 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 2294741 previous similar messages [Wed Jul 4 05:30:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 05:30:38 2018] Lustre: Skipped 2452828 previous similar messages [Wed Jul 4 05:30:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 05:30:38 2018] Lustre: Skipped 2452828 previous similar messages [Wed Jul 4 05:30:38 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 05:30:38 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 2452827 previous similar messages [Wed Jul 4 05:40:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 05:40:38 2018] Lustre: Skipped 2285982 previous similar messages [Wed Jul 4 05:40:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 05:40:38 2018] Lustre: Skipped 2285982 previous similar messages [Wed Jul 4 05:40:38 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 05:40:38 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 2285983 previous similar messages [Wed Jul 4 05:50:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 05:50:38 2018] Lustre: Skipped 2328094 previous similar messages [Wed Jul 4 05:50:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 05:50:38 2018] Lustre: Skipped 2328094 previous similar messages [Wed Jul 4 05:50:38 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 05:50:38 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 2328094 previous similar messages [Wed Jul 4 06:00:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 06:00:38 2018] Lustre: Skipped 2289472 previous similar messages [Wed Jul 4 06:00:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 06:00:38 2018] Lustre: Skipped 2289472 previous similar messages [Wed Jul 4 06:00:38 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 06:00:38 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 2289472 previous similar messages [Wed Jul 4 06:10:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 06:10:38 2018] Lustre: Skipped 2324970 previous similar messages [Wed Jul 4 06:10:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 06:10:38 2018] Lustre: Skipped 2324970 previous similar messages [Wed Jul 4 06:10:38 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 06:10:38 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 2324970 previous similar messages [Wed Jul 4 06:20:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 06:20:38 2018] Lustre: Skipped 2333113 previous similar messages [Wed Jul 4 06:20:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 06:20:38 2018] Lustre: Skipped 2333113 previous similar messages [Wed Jul 4 06:20:38 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 06:20:38 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 2333113 previous similar messages [Wed Jul 4 06:30:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 06:30:38 2018] Lustre: Skipped 2344836 previous similar messages [Wed Jul 4 06:30:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 06:30:38 2018] Lustre: Skipped 2344836 previous similar messages [Wed Jul 4 06:30:38 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 06:30:38 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 2344840 previous similar messages [Wed Jul 4 06:40:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 06:40:38 2018] Lustre: Skipped 2333989 previous similar messages [Wed Jul 4 06:40:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 06:40:38 2018] Lustre: Skipped 2333989 previous similar messages [Wed Jul 4 06:40:38 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 06:40:38 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2333984 previous similar messages [Wed Jul 4 06:50:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 06:50:38 2018] Lustre: Skipped 2286554 previous similar messages [Wed Jul 4 06:50:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 06:50:38 2018] Lustre: Skipped 2286554 previous similar messages [Wed Jul 4 06:50:38 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 06:50:38 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 2286586 previous similar messages [Wed Jul 4 07:00:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 07:00:38 2018] Lustre: Skipped 2319456 previous similar messages [Wed Jul 4 07:00:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 07:00:38 2018] Lustre: Skipped 2319457 previous similar messages [Wed Jul 4 07:00:38 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 07:00:38 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 2319459 previous similar messages [Wed Jul 4 07:10:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 07:10:38 2018] Lustre: Skipped 2308973 previous similar messages [Wed Jul 4 07:10:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 07:10:38 2018] Lustre: Skipped 2308974 previous similar messages [Wed Jul 4 07:10:38 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 07:10:38 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 2308969 previous similar messages [Wed Jul 4 07:20:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 07:20:38 2018] Lustre: Skipped 2308698 previous similar messages [Wed Jul 4 07:20:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 07:20:38 2018] Lustre: Skipped 2308698 previous similar messages [Wed Jul 4 07:20:38 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 07:20:38 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2308701 previous similar messages [Wed Jul 4 07:30:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 07:30:38 2018] Lustre: Skipped 2327776 previous similar messages [Wed Jul 4 07:30:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 07:30:38 2018] Lustre: Skipped 2327776 previous similar messages [Wed Jul 4 07:30:38 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 07:30:38 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 2327774 previous similar messages [Wed Jul 4 07:40:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 07:40:38 2018] Lustre: Skipped 2314947 previous similar messages [Wed Jul 4 07:40:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 07:40:38 2018] Lustre: Skipped 2314947 previous similar messages [Wed Jul 4 07:40:38 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 07:40:38 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 2314947 previous similar messages [Wed Jul 4 07:50:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 07:50:38 2018] Lustre: Skipped 2264430 previous similar messages [Wed Jul 4 07:50:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 07:50:38 2018] Lustre: Skipped 2264430 previous similar messages [Wed Jul 4 07:50:38 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 07:50:38 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 2264400 previous similar messages [Wed Jul 4 08:00:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 08:00:38 2018] Lustre: Skipped 2383936 previous similar messages [Wed Jul 4 08:00:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 08:00:38 2018] Lustre: Skipped 2383936 previous similar messages [Wed Jul 4 08:00:38 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 08:00:38 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 2384060 previous similar messages [Wed Jul 4 08:10:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 08:10:38 2018] Lustre: Skipped 2269647 previous similar messages [Wed Jul 4 08:10:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 08:10:38 2018] Lustre: Skipped 2269647 previous similar messages [Wed Jul 4 08:10:38 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 08:10:38 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 2269677 previous similar messages [Wed Jul 4 08:20:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 08:20:38 2018] Lustre: Skipped 2299894 previous similar messages [Wed Jul 4 08:20:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 08:20:38 2018] Lustre: Skipped 2299894 previous similar messages [Wed Jul 4 08:20:38 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 08:20:38 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 2299888 previous similar messages [Wed Jul 4 08:30:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 08:30:38 2018] Lustre: Skipped 2352773 previous similar messages [Wed Jul 4 08:30:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 08:30:38 2018] Lustre: Skipped 2352773 previous similar messages [Wed Jul 4 08:30:38 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 08:30:38 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 2352779 previous similar messages [Wed Jul 4 08:40:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 08:40:38 2018] Lustre: Skipped 1863332 previous similar messages [Wed Jul 4 08:40:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 08:40:38 2018] Lustre: Skipped 1863332 previous similar messages [Wed Jul 4 08:40:38 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 08:40:38 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 1863335 previous similar messages [Wed Jul 4 08:50:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 08:50:38 2018] Lustre: Skipped 2347757 previous similar messages [Wed Jul 4 08:50:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 08:50:38 2018] Lustre: Skipped 2347757 previous similar messages [Wed Jul 4 08:50:38 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 08:50:38 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 2347714 previous similar messages [Wed Jul 4 09:00:38 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 09:00:38 2018] Lustre: Skipped 2360148 previous similar messages [Wed Jul 4 09:00:38 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 09:00:38 2018] Lustre: Skipped 2360148 previous similar messages [Wed Jul 4 09:00:39 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 09:00:39 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 2360191 previous similar messages [Wed Jul 4 09:10:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 09:10:39 2018] Lustre: Skipped 2335458 previous similar messages [Wed Jul 4 09:10:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 09:10:39 2018] Lustre: Skipped 2335458 previous similar messages [Wed Jul 4 09:10:39 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 09:10:39 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 2335460 previous similar messages [Wed Jul 4 09:20:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 09:20:39 2018] Lustre: Skipped 2307040 previous similar messages [Wed Jul 4 09:20:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 09:20:39 2018] Lustre: Skipped 2307040 previous similar messages [Wed Jul 4 09:20:39 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 09:20:39 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 2307029 previous similar messages [Wed Jul 4 09:30:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 09:30:39 2018] Lustre: Skipped 2358994 previous similar messages [Wed Jul 4 09:30:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 09:30:39 2018] Lustre: Skipped 2358994 previous similar messages [Wed Jul 4 09:30:39 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 09:30:39 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 2359000 previous similar messages [Wed Jul 4 09:40:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 09:40:39 2018] Lustre: Skipped 1382829 previous similar messages [Wed Jul 4 09:40:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 09:40:39 2018] Lustre: Skipped 1382829 previous similar messages [Wed Jul 4 09:40:39 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 09:40:39 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 1382813 previous similar messages [Wed Jul 4 09:50:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 09:50:39 2018] Lustre: Skipped 2324768 previous similar messages [Wed Jul 4 09:50:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 09:50:39 2018] Lustre: Skipped 2324768 previous similar messages [Wed Jul 4 09:50:39 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 09:50:39 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 2324751 previous similar messages [Wed Jul 4 10:00:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 10:00:39 2018] Lustre: Skipped 1848221 previous similar messages [Wed Jul 4 10:00:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 10:00:39 2018] Lustre: Skipped 1848221 previous similar messages [Wed Jul 4 10:00:39 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 10:00:39 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 1848253 previous similar messages [Wed Jul 4 10:10:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 10:10:39 2018] Lustre: Skipped 1829464 previous similar messages [Wed Jul 4 10:10:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 10:10:39 2018] Lustre: Skipped 1829464 previous similar messages [Wed Jul 4 10:10:39 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 10:10:39 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 1829462 previous similar messages [Wed Jul 4 10:21:10 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 10:21:10 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 1063751 previous similar messages [Wed Jul 4 10:21:10 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 10:21:10 2018] Lustre: Skipped 1063910 previous similar messages [Wed Jul 4 10:21:10 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 10:21:10 2018] Lustre: Skipped 1063910 previous similar messages [Wed Jul 4 10:31:10 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 10:31:10 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 909347 previous similar messages [Wed Jul 4 10:31:10 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 10:31:10 2018] Lustre: Skipped 909347 previous similar messages [Wed Jul 4 10:31:10 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 10:31:10 2018] Lustre: Skipped 909347 previous similar messages [Wed Jul 4 10:41:10 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff88046afada00 x1604933427887200/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:308/0 lens 568/440 e 0 to 0 dl 1530664658 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 10:41:10 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88046afae600 x1604933427887200/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:308/0 lens 568/440 e 0 to 0 dl 1530664658 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 10:41:10 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 6 previous similar messages [Wed Jul 4 10:41:10 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 10:41:10 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 4 previous similar messages [Wed Jul 4 10:41:10 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 10:41:10 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 4400418 previous similar messages [Wed Jul 4 10:41:10 2018] Lustre: lustre-MDT0000: Client 4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad (at 172.16.229.39@o2ib) reconnecting [Wed Jul 4 10:41:10 2018] Lustre: Skipped 4400446 previous similar messages [Wed Jul 4 10:41:10 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Wed Jul 4 10:41:10 2018] Lustre: Skipped 4399791 previous similar messages [Wed Jul 4 10:41:13 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880a40f34500 x1604933428240880/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:311/0 lens 568/440 e 0 to 0 dl 1530664661 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 10:41:13 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffd2fd050 x1604933428240880/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:311/0 lens 568/440 e 0 to 0 dl 1530664661 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 10:41:13 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 3 previous similar messages [Wed Jul 4 10:41:13 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Wed Jul 4 10:41:16 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 10:41:16 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 1 previous similar message [Wed Jul 4 10:41:20 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff88041f4df500 x1604933429034736/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:318/0 lens 568/440 e 0 to 0 dl 1530664668 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 10:41:20 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880cd3610600 x1604933429034736/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:318/0 lens 568/440 e 0 to 0 dl 1530664668 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 10:41:20 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 3 previous similar messages [Wed Jul 4 10:41:20 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 10:41:20 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Wed Jul 4 10:41:25 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 10:41:31 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe316850 x1604933430233360/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:329/0 lens 568/440 e 0 to 0 dl 1530664679 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 10:41:31 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8803ac68b000 x1604933430233360/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:329/0 lens 568/440 e 0 to 0 dl 1530664679 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 10:41:31 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 3 previous similar messages [Wed Jul 4 10:41:38 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 10:41:38 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 1 previous similar message [Wed Jul 4 10:41:55 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8807b52c4200 x1604933432852512/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:353/0 lens 568/440 e 0 to 0 dl 1530664703 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 10:41:55 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880a2da12400 x1604933432852512/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:353/0 lens 568/440 e 0 to 0 dl 1530664703 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 10:41:55 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 5 previous similar messages [Wed Jul 4 10:41:55 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 2 previous similar messages [Wed Jul 4 10:42:42 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe311850 x1604933438107072/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:400/0 lens 568/440 e 0 to 0 dl 1530664750 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 10:42:42 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880a5f321800 x1604933438107072/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:400/0 lens 568/440 e 0 to 0 dl 1530664750 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 10:42:42 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 1 previous similar message [Wed Jul 4 10:42:42 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 10:42:42 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 2 previous similar messages [Wed Jul 4 10:43:24 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 10:43:24 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 4 previous similar messages [Wed Jul 4 10:45:04 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88012a1f2400 x1604933453721600/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:579/0 lens 568/440 e 0 to 0 dl 1530664929 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 10:45:04 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 10:45:04 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 2 previous similar messages [Wed Jul 4 10:45:04 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 24 previous similar messages [Wed Jul 4 10:45:10 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880269e9a700 x1604933454394656/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:585/0 lens 568/440 e 0 to 0 dl 1530664935 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 10:45:10 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 15 previous similar messages [Wed Jul 4 10:48:29 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe2f7850 x1604933474661360/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:29/0 lens 568/440 e 0 to 0 dl 1530665134 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 10:48:29 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffe317850 x1604933474661360/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:29/0 lens 568/440 e 0 to 0 dl 1530665134 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 10:48:29 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 38 previous similar messages [Wed Jul 4 10:48:29 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 19 previous similar messages [Wed Jul 4 10:48:30 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 10:48:30 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 17 previous similar messages [Wed Jul 4 10:51:10 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 10:51:10 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 6690324 previous similar messages [Wed Jul 4 10:51:10 2018] Lustre: lustre-MDT0000: Client 4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad (at 172.16.229.39@o2ib) reconnecting [Wed Jul 4 10:51:10 2018] Lustre: Skipped 6689956 previous similar messages [Wed Jul 4 10:51:10 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Wed Jul 4 10:51:10 2018] Lustre: Skipped 6688749 previous similar messages [Wed Jul 4 10:53:30 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2f9450 x1604933508466256/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:330/0 lens 568/440 e 0 to 0 dl 1530665435 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 10:53:30 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8802dcc4ce00 x1604933508466256/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:330/0 lens 568/440 e 0 to 0 dl 1530665435 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 10:53:30 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 130 previous similar messages [Wed Jul 4 10:53:30 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 72 previous similar messages [Wed Jul 4 10:53:31 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 10:53:31 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 58 previous similar messages [Wed Jul 4 11:01:10 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 11:01:10 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 6647369 previous similar messages [Wed Jul 4 11:01:10 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 11:01:10 2018] Lustre: Skipped 6646631 previous similar messages [Wed Jul 4 11:01:10 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 11:01:10 2018] Lustre: Skipped 6645206 previous similar messages [Wed Jul 4 11:03:30 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffe315850 x1604933574427744/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:174/0 lens 568/440 e 0 to 0 dl 1530666034 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 11:03:30 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 321 previous similar messages [Wed Jul 4 11:03:34 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe312050 x1604933575030224/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:179/0 lens 568/440 e 0 to 0 dl 1530666039 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 11:03:34 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 11:03:34 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 148 previous similar messages [Wed Jul 4 11:03:34 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 167 previous similar messages [Wed Jul 4 11:11:10 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 11:11:10 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 4989085 previous similar messages [Wed Jul 4 11:11:10 2018] Lustre: lustre-MDT0000: Client 4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad (at 172.16.229.39@o2ib) reconnecting [Wed Jul 4 11:11:10 2018] Lustre: Skipped 4988986 previous similar messages [Wed Jul 4 11:11:10 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Wed Jul 4 11:11:10 2018] Lustre: Skipped 4988218 previous similar messages [Wed Jul 4 11:13:30 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88049eb76600 x1604933630268752/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:19/0 lens 568/440 e 0 to 0 dl 1530666634 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 11:13:30 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 69 previous similar messages [Wed Jul 4 11:13:36 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880b8ca47500 x1604933631076576/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:26/0 lens 568/440 e 0 to 0 dl 1530666641 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 11:13:36 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 11:13:36 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 30 previous similar messages [Wed Jul 4 11:13:36 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 40 previous similar messages [Wed Jul 4 11:21:10 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 11:21:10 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 5985593 previous similar messages [Wed Jul 4 11:21:10 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 11:21:10 2018] Lustre: Skipped 5984972 previous similar messages [Wed Jul 4 11:21:10 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 11:21:10 2018] Lustre: Skipped 5983744 previous similar messages [Wed Jul 4 11:26:10 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe316050 x1604933684552416/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:743/0 lens 568/440 e 0 to 0 dl 1530667358 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 11:26:10 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8809e978ef00 x1604933684552416/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:743/0 lens 568/440 e 0 to 0 dl 1530667358 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 11:26:10 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 30 previous similar messages [Wed Jul 4 11:26:10 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 11:26:10 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 12 previous similar messages [Wed Jul 4 11:26:10 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 15 previous similar messages [Wed Jul 4 11:31:10 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 11:31:10 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 6063635 previous similar messages [Wed Jul 4 11:31:10 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 11:31:10 2018] Lustre: Skipped 6063140 previous similar messages [Wed Jul 4 11:31:10 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 11:31:10 2018] Lustre: Skipped 6060825 previous similar messages [Wed Jul 4 11:39:03 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880c587c0c00 x1604933765334816/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:5/0 lens 568/440 e 0 to 0 dl 1530668130 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 11:39:03 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880c587c0f00 x1604933765334816/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:5/0 lens 568/440 e 0 to 0 dl 1530668130 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 11:39:03 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 57 previous similar messages [Wed Jul 4 11:39:03 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Wed Jul 4 11:39:03 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 26 previous similar messages [Wed Jul 4 11:39:03 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 33 previous similar messages [Wed Jul 4 11:40:35 2018] LustreError: 25630:0:(ldlm_lockd.c:690:ldlm_handle_ast_error()) ### client (nid 172.16.229.39@o2ib) returned error from blocking AST (req@ffff880390f34500 x1601781824716272 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: ffff880d83d15580/0x88fac5bdeea6f1b lrc: 4/0,0 mode: PR/PR res: [0x2000131ba:0x1c72:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 172.16.229.39@o2ib remote: 0x34f47352b46218a4 expref: 3362 pid: 42073 timeout: 3091514 lvb_type: 0 [Wed Jul 4 11:40:35 2018] LustreError: 138-a: lustre-MDT0000: A client on nid 172.16.229.39@o2ib was evicted due to a lock blocking callback time out: rc -107 [Wed Jul 4 11:41:10 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 11:41:10 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 6416069 previous similar messages [Wed Jul 4 11:41:10 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 11:41:10 2018] Lustre: Skipped 6415575 previous similar messages [Wed Jul 4 11:41:10 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 11:41:10 2018] Lustre: Skipped 6414411 previous similar messages [Wed Jul 4 11:44:33 2018] LustreError: 156579:0:(ldlm_lockd.c:690:ldlm_handle_ast_error()) ### client (nid 172.16.229.39@o2ib) returned error from blocking AST (req@ffff881bf939c800 x1601781824731872 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: ffff880445352400/0x88fac5be0360cf9 lrc: 4/0,0 mode: PR/PR res: [0x20001317f:0x14b85:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 172.16.229.39@o2ib remote: 0x34f47352b4621e38 expref: 3230 pid: 42073 timeout: 3090953 lvb_type: 0 [Wed Jul 4 11:44:33 2018] LustreError: 138-a: lustre-MDT0000: A client on nid 172.16.229.39@o2ib was evicted due to a lock blocking callback time out: rc -107 [Wed Jul 4 11:44:34 2018] LustreError: 4382:0:(ldlm_lib.c:3223:target_bulk_io()) @@@ Eviction on bulk READ req@ffff880105c64800 x1604933803183280/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:336/0 lens 568/440 e 0 to 0 dl 1530668461 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 11:51:52 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 11:51:52 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 2233386 previous similar messages [Wed Jul 4 11:51:52 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 11:51:52 2018] Lustre: Skipped 2233182 previous similar messages [Wed Jul 4 11:51:52 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 11:51:52 2018] Lustre: Skipped 2232930 previous similar messages [Wed Jul 4 11:56:29 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880d55fb4500 x1604325891377104/t0(0) o37->74dc27c9-7eb7-6236-850f-6507aace669b@172.16.230.55@o2ib:334/0 lens 568/440 e 0 to 0 dl 1530669214 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 11:56:29 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffe2f6850 x1604325891377104/t0(0) o37->74dc27c9-7eb7-6236-850f-6507aace669b@172.16.230.55@o2ib:334/0 lens 568/440 e 0 to 0 dl 1530669214 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 11:56:29 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 55 previous similar messages [Wed Jul 4 11:56:29 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 30 previous similar messages [Wed Jul 4 12:02:15 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 12:02:15 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 1060725 previous similar messages [Wed Jul 4 12:02:15 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 12:02:15 2018] Lustre: Skipped 1060725 previous similar messages [Wed Jul 4 12:02:15 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 12:02:15 2018] Lustre: Skipped 1060725 previous similar messages [Wed Jul 4 12:12:15 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 12:12:15 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 1372421 previous similar messages [Wed Jul 4 12:12:15 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 12:12:15 2018] Lustre: Skipped 1372421 previous similar messages [Wed Jul 4 12:12:15 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 12:12:15 2018] Lustre: Skipped 1372421 previous similar messages [Wed Jul 4 12:23:17 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 12:23:17 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 1197604 previous similar messages [Wed Jul 4 12:23:17 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 12:23:17 2018] Lustre: Skipped 1197604 previous similar messages [Wed Jul 4 12:23:17 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 12:23:17 2018] Lustre: Skipped 1197604 previous similar messages [Wed Jul 4 12:33:50 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 12:33:50 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 614086 previous similar messages [Wed Jul 4 12:33:51 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 12:33:51 2018] Lustre: Skipped 614086 previous similar messages [Wed Jul 4 12:33:51 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 12:33:51 2018] Lustre: Skipped 614086 previous similar messages [Wed Jul 4 12:45:30 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 12:45:30 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 1362392 previous similar messages [Wed Jul 4 12:45:30 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 12:45:30 2018] Lustre: Skipped 1362392 previous similar messages [Wed Jul 4 12:45:30 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 12:45:30 2018] Lustre: Skipped 1362392 previous similar messages [Wed Jul 4 12:56:02 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 12:56:02 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 149638 previous similar messages [Wed Jul 4 12:56:02 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 12:56:02 2018] Lustre: Skipped 149638 previous similar messages [Wed Jul 4 12:56:02 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 12:56:02 2018] Lustre: Skipped 149638 previous similar messages [Wed Jul 4 13:06:02 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 13:06:02 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 926352 previous similar messages [Wed Jul 4 13:06:02 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 13:06:02 2018] Lustre: Skipped 926352 previous similar messages [Wed Jul 4 13:06:02 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 13:06:02 2018] Lustre: Skipped 926352 previous similar messages [Wed Jul 4 13:16:34 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 13:16:34 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 1547543 previous similar messages [Wed Jul 4 13:16:34 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 13:16:34 2018] Lustre: Skipped 1547543 previous similar messages [Wed Jul 4 13:16:34 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 13:16:34 2018] Lustre: Skipped 1547543 previous similar messages [Wed Jul 4 13:27:32 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 13:27:32 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 1187938 previous similar messages [Wed Jul 4 13:27:32 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 13:27:32 2018] Lustre: Skipped 1187938 previous similar messages [Wed Jul 4 13:27:32 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 13:27:32 2018] Lustre: Skipped 1187938 previous similar messages [Wed Jul 4 13:37:32 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 13:37:32 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 1396659 previous similar messages [Wed Jul 4 13:37:32 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 13:37:32 2018] Lustre: Skipped 1396659 previous similar messages [Wed Jul 4 13:37:32 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 13:37:32 2018] Lustre: Skipped 1396659 previous similar messages [Wed Jul 4 13:47:32 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 13:47:32 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 936410 previous similar messages [Wed Jul 4 13:47:32 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 13:47:32 2018] Lustre: Skipped 936410 previous similar messages [Wed Jul 4 13:47:32 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 13:47:32 2018] Lustre: Skipped 936410 previous similar messages [Wed Jul 4 13:58:39 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 13:58:39 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 762872 previous similar messages [Wed Jul 4 13:58:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 13:58:39 2018] Lustre: Skipped 762872 previous similar messages [Wed Jul 4 13:58:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 13:58:39 2018] Lustre: Skipped 762872 previous similar messages [Wed Jul 4 14:02:32 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffd2fa850 x1603047438751440/t0(0) o37->66471b3c-6a3e-724d-5030-ee8252fcfcd2@172.16.230.87@o2ib:309/0 lens 568/440 e 0 to 0 dl 1530676739 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 14:02:32 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.230.87@o2ib [Wed Jul 4 14:02:32 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 25 previous similar messages [Wed Jul 4 14:02:32 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 2 previous similar messages [Wed Jul 4 14:02:33 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe316c50 x1603047438818320/t0(0) o37->66471b3c-6a3e-724d-5030-ee8252fcfcd2@172.16.230.87@o2ib:310/0 lens 568/440 e 0 to 0 dl 1530676740 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 14:02:33 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 2 previous similar messages [Wed Jul 4 14:08:39 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 14:08:39 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 4534589 previous similar messages [Wed Jul 4 14:08:39 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 14:08:39 2018] Lustre: Skipped 3503233 previous similar messages [Wed Jul 4 14:08:39 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 14:08:39 2018] Lustre: Skipped 3502882 previous similar messages [Wed Jul 4 14:18:39 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 14:18:39 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 7321057 previous similar messages [Wed Jul 4 14:18:39 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 14:18:39 2018] Lustre: Skipped 5026988 previous similar messages [Wed Jul 4 14:18:39 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 14:18:39 2018] Lustre: Skipped 5026269 previous similar messages [Wed Jul 4 14:28:39 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 14:28:39 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 7296549 previous similar messages [Wed Jul 4 14:28:39 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 14:28:39 2018] Lustre: Skipped 4695718 previous similar messages [Wed Jul 4 14:28:39 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 14:28:39 2018] Lustre: Skipped 4695229 previous similar messages [Wed Jul 4 14:38:39 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 14:38:39 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 6977166 previous similar messages [Wed Jul 4 14:38:39 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 14:38:39 2018] Lustre: Skipped 4718086 previous similar messages [Wed Jul 4 14:38:39 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 14:38:39 2018] Lustre: Skipped 4717511 previous similar messages [Wed Jul 4 14:48:39 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 14:48:39 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 6999748 previous similar messages [Wed Jul 4 14:48:39 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 14:48:39 2018] Lustre: Skipped 4720354 previous similar messages [Wed Jul 4 14:48:39 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 14:48:39 2018] Lustre: Skipped 4719673 previous similar messages [Wed Jul 4 14:58:39 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 14:58:39 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 7587913 previous similar messages [Wed Jul 4 14:58:39 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 14:58:39 2018] Lustre: Skipped 5014355 previous similar messages [Wed Jul 4 14:58:39 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 14:58:39 2018] Lustre: Skipped 5013638 previous similar messages [Wed Jul 4 15:08:39 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 15:08:39 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 7459364 previous similar messages [Wed Jul 4 15:08:39 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 15:08:39 2018] Lustre: Skipped 5009097 previous similar messages [Wed Jul 4 15:08:39 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 15:08:39 2018] Lustre: Skipped 5008175 previous similar messages [Wed Jul 4 15:18:39 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 15:18:39 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 6648432 previous similar messages [Wed Jul 4 15:18:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 15:18:39 2018] Lustre: Skipped 4433314 previous similar messages [Wed Jul 4 15:18:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 15:18:39 2018] Lustre: Skipped 4432717 previous similar messages [Wed Jul 4 15:28:39 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 15:28:39 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 6730070 previous similar messages [Wed Jul 4 15:28:39 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 15:28:39 2018] Lustre: Skipped 4421757 previous similar messages [Wed Jul 4 15:28:39 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 15:28:39 2018] Lustre: Skipped 4421203 previous similar messages [Wed Jul 4 15:38:39 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 15:38:39 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 6945465 previous similar messages [Wed Jul 4 15:38:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 15:38:39 2018] Lustre: Skipped 4704858 previous similar messages [Wed Jul 4 15:38:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 15:38:39 2018] Lustre: Skipped 4704228 previous similar messages [Wed Jul 4 15:48:39 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 15:48:39 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 7111458 previous similar messages [Wed Jul 4 15:48:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 15:48:39 2018] Lustre: Skipped 4376839 previous similar messages [Wed Jul 4 15:48:39 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 15:48:39 2018] Lustre: Skipped 4376413 previous similar messages [Wed Jul 4 15:58:39 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 15:58:39 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 8932333 previous similar messages [Wed Jul 4 15:58:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 15:58:39 2018] Lustre: Skipped 5843292 previous similar messages [Wed Jul 4 15:58:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 15:58:39 2018] Lustre: Skipped 5842574 previous similar messages [Wed Jul 4 16:08:39 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 16:08:39 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 7908638 previous similar messages [Wed Jul 4 16:08:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 16:08:39 2018] Lustre: Skipped 4938268 previous similar messages [Wed Jul 4 16:08:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 16:08:39 2018] Lustre: Skipped 4937835 previous similar messages [Wed Jul 4 16:18:39 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 16:18:39 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 7434285 previous similar messages [Wed Jul 4 16:18:39 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 16:18:39 2018] Lustre: Skipped 5260204 previous similar messages [Wed Jul 4 16:18:39 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 16:18:39 2018] Lustre: Skipped 5259248 previous similar messages [Wed Jul 4 16:28:39 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 16:28:39 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 6752710 previous similar messages [Wed Jul 4 16:28:39 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 16:28:39 2018] Lustre: Skipped 4769021 previous similar messages [Wed Jul 4 16:28:39 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 16:28:39 2018] Lustre: Skipped 4768561 previous similar messages [Wed Jul 4 16:38:39 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 16:38:39 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 7397791 previous similar messages [Wed Jul 4 16:38:39 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 16:38:39 2018] Lustre: Skipped 5100324 previous similar messages [Wed Jul 4 16:38:39 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 16:38:39 2018] Lustre: Skipped 5099513 previous similar messages [Wed Jul 4 16:48:39 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 16:48:39 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 6919227 previous similar messages [Wed Jul 4 16:48:39 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 16:48:39 2018] Lustre: Skipped 4600658 previous similar messages [Wed Jul 4 16:48:39 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 16:48:39 2018] Lustre: Skipped 4600089 previous similar messages [Wed Jul 4 16:58:39 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 16:58:39 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 7729077 previous similar messages [Wed Jul 4 16:58:40 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 16:58:40 2018] Lustre: Skipped 5166934 previous similar messages [Wed Jul 4 16:58:40 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 16:58:40 2018] Lustre: Skipped 5166208 previous similar messages [Wed Jul 4 17:08:40 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 17:08:40 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 7754144 previous similar messages [Wed Jul 4 17:08:40 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 17:08:40 2018] Lustre: Skipped 5166450 previous similar messages [Wed Jul 4 17:08:40 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 17:08:40 2018] Lustre: Skipped 5165851 previous similar messages [Wed Jul 4 17:18:40 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 17:18:40 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 7559437 previous similar messages [Wed Jul 4 17:18:40 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 17:18:40 2018] Lustre: Skipped 5325519 previous similar messages [Wed Jul 4 17:18:40 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 17:18:40 2018] Lustre: Skipped 5324766 previous similar messages [Wed Jul 4 17:18:47 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff88092ae31e00 x1604326841357168/t0(0) o37->74dc27c9-7eb7-6236-850f-6507aace669b@172.16.230.55@o2ib:4/0 lens 568/440 e 0 to 0 dl 1530688514 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Jul 4 17:18:47 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880bec7d9500 x1604326841357168/t0(0) o37->74dc27c9-7eb7-6236-850f-6507aace669b@172.16.230.55@o2ib:4/0 lens 568/440 e 0 to 0 dl 1530688514 ref 1 fl Interpret:/2/0 rc 0/0 [Wed Jul 4 17:18:47 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 11 previous similar messages [Wed Jul 4 17:18:47 2018] LustreError: 159530:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Wed Jul 4 17:28:40 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 17:28:40 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 9162695 previous similar messages [Wed Jul 4 17:28:40 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 17:28:40 2018] Lustre: Skipped 5863944 previous similar messages [Wed Jul 4 17:28:40 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 17:28:40 2018] Lustre: Skipped 5863237 previous similar messages [Wed Jul 4 17:38:40 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 17:38:40 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 6463671 previous similar messages [Wed Jul 4 17:38:40 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 17:38:40 2018] Lustre: Skipped 4292942 previous similar messages [Wed Jul 4 17:38:40 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 17:38:40 2018] Lustre: Skipped 4292458 previous similar messages [Wed Jul 4 17:48:40 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 17:48:40 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 6566699 previous similar messages [Wed Jul 4 17:48:40 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 17:48:40 2018] Lustre: Skipped 4277643 previous similar messages [Wed Jul 4 17:48:40 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 17:48:40 2018] Lustre: Skipped 4277221 previous similar messages [Wed Jul 4 17:58:40 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 17:58:40 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 8830292 previous similar messages [Wed Jul 4 17:58:40 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 17:58:40 2018] Lustre: Skipped 5801235 previous similar messages [Wed Jul 4 17:58:40 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 17:58:40 2018] Lustre: Skipped 5800471 previous similar messages [Wed Jul 4 18:08:40 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 18:08:40 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 7737714 previous similar messages [Wed Jul 4 18:08:40 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 18:08:40 2018] Lustre: Skipped 4830507 previous similar messages [Wed Jul 4 18:08:40 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 18:08:40 2018] Lustre: Skipped 4829944 previous similar messages [Wed Jul 4 18:18:40 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 18:18:40 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 7882842 previous similar messages [Wed Jul 4 18:18:40 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 18:18:40 2018] Lustre: Skipped 5301919 previous similar messages [Wed Jul 4 18:18:40 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 18:18:40 2018] Lustre: Skipped 5302832 previous similar messages [Wed Jul 4 18:28:40 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 18:28:40 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 7644329 previous similar messages [Wed Jul 4 18:28:40 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 18:28:40 2018] Lustre: Skipped 5127698 previous similar messages [Wed Jul 4 18:28:40 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 18:28:40 2018] Lustre: Skipped 5126988 previous similar messages [Wed Jul 4 18:38:40 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 18:38:40 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 7007828 previous similar messages [Wed Jul 4 18:38:40 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 18:38:40 2018] Lustre: Skipped 4671604 previous similar messages [Wed Jul 4 18:38:40 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 18:38:40 2018] Lustre: Skipped 4670881 previous similar messages [Wed Jul 4 18:48:40 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 18:48:40 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 6200318 previous similar messages [Wed Jul 4 18:48:40 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 18:48:40 2018] Lustre: Skipped 4237024 previous similar messages [Wed Jul 4 18:48:40 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 18:48:40 2018] Lustre: Skipped 4236370 previous similar messages [Wed Jul 4 18:58:40 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 18:58:40 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 6855266 previous similar messages [Wed Jul 4 18:58:40 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 18:58:40 2018] Lustre: Skipped 4287829 previous similar messages [Wed Jul 4 18:58:40 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 18:58:40 2018] Lustre: Skipped 4287269 previous similar messages [Wed Jul 4 19:08:40 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 19:08:40 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 7444467 previous similar messages [Wed Jul 4 19:08:40 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 19:08:40 2018] Lustre: Skipped 4881004 previous similar messages [Wed Jul 4 19:08:40 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 19:08:40 2018] Lustre: Skipped 4880295 previous similar messages [Wed Jul 4 19:18:40 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 19:18:40 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 7602814 previous similar messages [Wed Jul 4 19:18:40 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 19:18:40 2018] Lustre: Skipped 5164108 previous similar messages [Wed Jul 4 19:18:40 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 19:18:40 2018] Lustre: Skipped 5163371 previous similar messages [Wed Jul 4 19:28:40 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 19:28:40 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 6796165 previous similar messages [Wed Jul 4 19:28:40 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 19:28:40 2018] Lustre: Skipped 4519902 previous similar messages [Wed Jul 4 19:28:40 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 19:28:40 2018] Lustre: Skipped 4519292 previous similar messages [Wed Jul 4 19:38:40 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 19:38:40 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 7280365 previous similar messages [Wed Jul 4 19:38:40 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 19:38:40 2018] Lustre: Skipped 4994836 previous similar messages [Wed Jul 4 19:38:40 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 19:38:40 2018] Lustre: Skipped 4994230 previous similar messages [Wed Jul 4 19:48:40 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 19:48:40 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 7307575 previous similar messages [Wed Jul 4 19:48:40 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 19:48:40 2018] Lustre: Skipped 5027696 previous similar messages [Wed Jul 4 19:48:40 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 19:48:40 2018] Lustre: Skipped 5027107 previous similar messages [Wed Jul 4 19:58:40 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 19:58:40 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 7878865 previous similar messages [Wed Jul 4 19:58:40 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 19:58:40 2018] Lustre: Skipped 5327858 previous similar messages [Wed Jul 4 19:58:40 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 19:58:40 2018] Lustre: Skipped 5327007 previous similar messages [Wed Jul 4 20:08:40 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 20:08:40 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 7021587 previous similar messages [Wed Jul 4 20:08:40 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 20:08:40 2018] Lustre: Skipped 4527114 previous similar messages [Wed Jul 4 20:08:40 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 20:08:40 2018] Lustre: Skipped 4526453 previous similar messages [Wed Jul 4 20:18:40 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 20:18:40 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 8434435 previous similar messages [Wed Jul 4 20:18:40 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 20:18:40 2018] Lustre: Skipped 5576385 previous similar messages [Wed Jul 4 20:18:40 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 20:18:40 2018] Lustre: Skipped 5575639 previous similar messages [Wed Jul 4 20:28:40 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 20:28:40 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 7490493 previous similar messages [Wed Jul 4 20:28:40 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 20:28:40 2018] Lustre: Skipped 4687472 previous similar messages [Wed Jul 4 20:28:40 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 20:28:40 2018] Lustre: Skipped 4686995 previous similar messages [Wed Jul 4 20:38:40 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 20:38:40 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 7302763 previous similar messages [Wed Jul 4 20:38:41 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 20:38:41 2018] Lustre: Skipped 4992859 previous similar messages [Wed Jul 4 20:38:41 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 20:38:41 2018] Lustre: Skipped 4992159 previous similar messages [Wed Jul 4 20:48:41 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 20:48:41 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 6878962 previous similar messages [Wed Jul 4 20:48:41 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 20:48:41 2018] Lustre: Skipped 4563889 previous similar messages [Wed Jul 4 20:48:41 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 20:48:41 2018] Lustre: Skipped 4563312 previous similar messages [Wed Jul 4 20:58:41 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 20:58:41 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 8044562 previous similar messages [Wed Jul 4 20:58:41 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 20:58:41 2018] Lustre: Skipped 5342650 previous similar messages [Wed Jul 4 20:58:41 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 20:58:41 2018] Lustre: Skipped 5342100 previous similar messages [Wed Jul 4 21:08:41 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 21:08:41 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 8607652 previous similar messages [Wed Jul 4 21:08:41 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 21:08:41 2018] Lustre: Skipped 5578516 previous similar messages [Wed Jul 4 21:08:41 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 21:08:41 2018] Lustre: Skipped 5577832 previous similar messages [Wed Jul 4 21:18:41 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 21:18:41 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 7543801 previous similar messages [Wed Jul 4 21:18:41 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 21:18:41 2018] Lustre: Skipped 5133820 previous similar messages [Wed Jul 4 21:18:41 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 21:18:41 2018] Lustre: Skipped 5132969 previous similar messages [Wed Jul 4 21:28:41 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 21:28:41 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 7929277 previous similar messages [Wed Jul 4 21:28:41 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 21:28:41 2018] Lustre: Skipped 5343778 previous similar messages [Wed Jul 4 21:28:41 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 21:28:41 2018] Lustre: Skipped 5343034 previous similar messages [Wed Jul 4 21:38:41 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 21:38:41 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 7961549 previous similar messages [Wed Jul 4 21:38:41 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 21:38:41 2018] Lustre: Skipped 5333973 previous similar messages [Wed Jul 4 21:38:41 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 21:38:41 2018] Lustre: Skipped 5333293 previous similar messages [Wed Jul 4 21:48:41 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 21:48:41 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 6614934 previous similar messages [Wed Jul 4 21:48:41 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 21:48:41 2018] Lustre: Skipped 4418491 previous similar messages [Wed Jul 4 21:48:41 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 21:48:41 2018] Lustre: Skipped 4418016 previous similar messages [Wed Jul 4 21:58:41 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 21:58:41 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 9219421 previous similar messages [Wed Jul 4 21:58:41 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 21:58:41 2018] Lustre: Skipped 5986065 previous similar messages [Wed Jul 4 21:58:41 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 21:58:41 2018] Lustre: Skipped 5985487 previous similar messages [Wed Jul 4 22:08:41 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 22:08:41 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 9082562 previous similar messages [Wed Jul 4 22:08:41 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 22:08:41 2018] Lustre: Skipped 5907298 previous similar messages [Wed Jul 4 22:08:41 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 22:08:41 2018] Lustre: Skipped 5906658 previous similar messages [Wed Jul 4 22:18:41 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 22:18:41 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 7848702 previous similar messages [Wed Jul 4 22:18:41 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 22:18:41 2018] Lustre: Skipped 5071711 previous similar messages [Wed Jul 4 22:18:41 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 22:18:41 2018] Lustre: Skipped 5071300 previous similar messages [Wed Jul 4 22:28:41 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 22:28:41 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 8414439 previous similar messages [Wed Jul 4 22:28:41 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 22:28:41 2018] Lustre: Skipped 5587022 previous similar messages [Wed Jul 4 22:28:41 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 22:28:41 2018] Lustre: Skipped 5586646 previous similar messages [Wed Jul 4 22:38:41 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 22:38:41 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 8753841 previous similar messages [Wed Jul 4 22:38:41 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 22:38:41 2018] Lustre: Skipped 5736002 previous similar messages [Wed Jul 4 22:38:41 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 22:38:41 2018] Lustre: Skipped 5735441 previous similar messages [Wed Jul 4 22:48:41 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 22:48:41 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) Skipped 7400690 previous similar messages [Wed Jul 4 22:48:41 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 22:48:41 2018] Lustre: Skipped 5017901 previous similar messages [Wed Jul 4 22:48:41 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 22:48:41 2018] Lustre: Skipped 5017312 previous similar messages [Wed Jul 4 22:58:41 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 22:58:41 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 7534826 previous similar messages [Wed Jul 4 22:58:41 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 22:58:41 2018] Lustre: Skipped 5065717 previous similar messages [Wed Jul 4 22:58:41 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 22:58:41 2018] Lustre: Skipped 5065136 previous similar messages [Wed Jul 4 23:08:41 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 23:08:41 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 7925662 previous similar messages [Wed Jul 4 23:08:41 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 23:08:41 2018] Lustre: Skipped 5311593 previous similar messages [Wed Jul 4 23:08:41 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 23:08:41 2018] Lustre: Skipped 5310808 previous similar messages [Wed Jul 4 23:18:41 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 23:18:41 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 7523963 previous similar messages [Wed Jul 4 23:18:41 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 23:18:41 2018] Lustre: Skipped 5121890 previous similar messages [Wed Jul 4 23:18:41 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 23:18:41 2018] Lustre: Skipped 5121229 previous similar messages [Wed Jul 4 23:28:41 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 23:28:41 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 8095114 previous similar messages [Wed Jul 4 23:28:41 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 23:28:41 2018] Lustre: Skipped 5189797 previous similar messages [Wed Jul 4 23:28:41 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 23:28:41 2018] Lustre: Skipped 5189341 previous similar messages [Wed Jul 4 23:38:41 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 23:38:41 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 7691537 previous similar messages [Wed Jul 4 23:38:41 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 23:38:41 2018] Lustre: Skipped 4984399 previous similar messages [Wed Jul 4 23:38:41 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 23:38:41 2018] Lustre: Skipped 4983987 previous similar messages [Wed Jul 4 23:48:41 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 23:48:41 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 8719515 previous similar messages [Wed Jul 4 23:48:41 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Wed Jul 4 23:48:41 2018] Lustre: Skipped 5726211 previous similar messages [Wed Jul 4 23:48:41 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Wed Jul 4 23:48:41 2018] Lustre: Skipped 5725627 previous similar messages [Wed Jul 4 23:58:41 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Wed Jul 4 23:58:41 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 9024304 previous similar messages [Wed Jul 4 23:58:41 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Wed Jul 4 23:58:41 2018] Lustre: Skipped 5652464 previous similar messages [Wed Jul 4 23:58:41 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Wed Jul 4 23:58:41 2018] Lustre: Skipped 5651919 previous similar messages [Thu Jul 5 00:08:41 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 00:08:41 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 8952619 previous similar messages [Thu Jul 5 00:08:41 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 00:08:41 2018] Lustre: Skipped 6177088 previous similar messages [Thu Jul 5 00:08:41 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 00:08:41 2018] Lustre: Skipped 6176363 previous similar messages [Thu Jul 5 00:18:41 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 00:18:41 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 9591013 previous similar messages [Thu Jul 5 00:18:41 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 00:18:41 2018] Lustre: Skipped 6152756 previous similar messages [Thu Jul 5 00:18:41 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 00:18:41 2018] Lustre: Skipped 6152035 previous similar messages [Thu Jul 5 00:28:41 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 00:28:41 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 9269819 previous similar messages [Thu Jul 5 00:28:42 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 00:28:42 2018] Lustre: Skipped 6008335 previous similar messages [Thu Jul 5 00:28:42 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 00:28:42 2018] Lustre: Skipped 6007634 previous similar messages [Thu Jul 5 00:38:42 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 00:38:42 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 9259770 previous similar messages [Thu Jul 5 00:38:42 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 00:38:42 2018] Lustre: Skipped 5977719 previous similar messages [Thu Jul 5 00:38:42 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 00:38:42 2018] Lustre: Skipped 5977030 previous similar messages [Thu Jul 5 00:48:42 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 00:48:42 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 8125270 previous similar messages [Thu Jul 5 00:48:42 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 00:48:42 2018] Lustre: Skipped 5206109 previous similar messages [Thu Jul 5 00:48:42 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 00:48:42 2018] Lustre: Skipped 5205693 previous similar messages [Thu Jul 5 00:58:42 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 00:58:42 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) Skipped 9644471 previous similar messages [Thu Jul 5 00:58:42 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 00:58:42 2018] Lustre: Skipped 6202103 previous similar messages [Thu Jul 5 00:58:42 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 00:58:42 2018] Lustre: Skipped 6201394 previous similar messages [Thu Jul 5 01:08:42 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 01:08:42 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 8638096 previous similar messages [Thu Jul 5 01:08:42 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 01:08:42 2018] Lustre: Skipped 5690299 previous similar messages [Thu Jul 5 01:08:42 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 01:08:42 2018] Lustre: Skipped 5689723 previous similar messages [Thu Jul 5 01:18:42 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 01:18:42 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 8276997 previous similar messages [Thu Jul 5 01:18:42 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 01:18:42 2018] Lustre: Skipped 5472048 previous similar messages [Thu Jul 5 01:18:42 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 01:18:42 2018] Lustre: Skipped 5471807 previous similar messages [Thu Jul 5 01:28:42 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 01:28:42 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 8530160 previous similar messages [Thu Jul 5 01:28:42 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 01:28:42 2018] Lustre: Skipped 5494343 previous similar messages [Thu Jul 5 01:28:42 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 01:28:42 2018] Lustre: Skipped 5493734 previous similar messages [Thu Jul 5 01:38:42 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 01:38:42 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 9304894 previous similar messages [Thu Jul 5 01:38:42 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 01:38:42 2018] Lustre: Skipped 6009694 previous similar messages [Thu Jul 5 01:38:42 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 01:38:42 2018] Lustre: Skipped 6009020 previous similar messages [Thu Jul 5 01:48:42 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 01:48:42 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 9286639 previous similar messages [Thu Jul 5 01:48:42 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 01:48:42 2018] Lustre: Skipped 6006392 previous similar messages [Thu Jul 5 01:48:42 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 01:48:42 2018] Lustre: Skipped 6005665 previous similar messages [Thu Jul 5 01:58:42 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 01:58:42 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 9562516 previous similar messages [Thu Jul 5 01:58:42 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 01:58:42 2018] Lustre: Skipped 6180959 previous similar messages [Thu Jul 5 01:58:42 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 01:58:42 2018] Lustre: Skipped 6180350 previous similar messages [Thu Jul 5 02:08:42 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 02:08:42 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 8662803 previous similar messages [Thu Jul 5 02:08:42 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 02:08:42 2018] Lustre: Skipped 5709928 previous similar messages [Thu Jul 5 02:08:42 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 02:08:42 2018] Lustre: Skipped 5709395 previous similar messages [Thu Jul 5 02:18:42 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 02:18:42 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 7570961 previous similar messages [Thu Jul 5 02:18:42 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 02:18:42 2018] Lustre: Skipped 5138932 previous similar messages [Thu Jul 5 02:18:42 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 02:18:42 2018] Lustre: Skipped 5137675 previous similar messages [Thu Jul 5 02:28:42 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 02:28:42 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 8737442 previous similar messages [Thu Jul 5 02:28:42 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 02:28:42 2018] Lustre: Skipped 5494610 previous similar messages [Thu Jul 5 02:28:42 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 02:28:42 2018] Lustre: Skipped 5493974 previous similar messages [Thu Jul 5 02:38:42 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 02:38:42 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) Skipped 8876746 previous similar messages [Thu Jul 5 02:38:42 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 02:38:42 2018] Lustre: Skipped 5582062 previous similar messages [Thu Jul 5 02:38:42 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 02:38:42 2018] Lustre: Skipped 5581511 previous similar messages [Thu Jul 5 02:48:42 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 02:48:42 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 8850854 previous similar messages [Thu Jul 5 02:48:42 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 02:48:42 2018] Lustre: Skipped 5548101 previous similar messages [Thu Jul 5 02:48:42 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 02:48:42 2018] Lustre: Skipped 5547487 previous similar messages [Thu Jul 5 02:59:34 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 02:59:34 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 02:59:34 2018] Lustre: Skipped 5897775 previous similar messages [Thu Jul 5 02:59:34 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 02:59:34 2018] Lustre: Skipped 5897079 previous similar messages [Thu Jul 5 02:59:34 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 9186994 previous similar messages [Thu Jul 5 03:09:34 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 03:09:34 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 03:09:34 2018] Lustre: Skipped 5374162 previous similar messages [Thu Jul 5 03:09:34 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 03:09:34 2018] Lustre: Skipped 5373731 previous similar messages [Thu Jul 5 03:09:34 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 8325276 previous similar messages [Thu Jul 5 03:19:34 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 03:19:34 2018] Lustre: Skipped 5302819 previous similar messages [Thu Jul 5 03:19:34 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 03:19:34 2018] Lustre: Skipped 5302080 previous similar messages [Thu Jul 5 03:19:34 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 03:19:34 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 8006254 previous similar messages [Thu Jul 5 03:30:54 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 03:30:54 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 9015592 previous similar messages [Thu Jul 5 03:30:54 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 03:30:54 2018] Lustre: Skipped 5832508 previous similar messages [Thu Jul 5 03:30:54 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 03:30:54 2018] Lustre: Skipped 5831863 previous similar messages [Thu Jul 5 03:40:54 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 03:40:54 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 7413090 previous similar messages [Thu Jul 5 03:40:54 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 03:40:54 2018] Lustre: Skipped 4881368 previous similar messages [Thu Jul 5 03:40:54 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 03:40:54 2018] Lustre: Skipped 4880731 previous similar messages [Thu Jul 5 03:50:54 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 03:50:54 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 7690615 previous similar messages [Thu Jul 5 03:50:54 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 03:50:54 2018] Lustre: Skipped 5207646 previous similar messages [Thu Jul 5 03:50:54 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 03:50:54 2018] Lustre: Skipped 5206737 previous similar messages [Thu Jul 5 04:00:54 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 04:00:54 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 9419149 previous similar messages [Thu Jul 5 04:00:54 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 04:00:54 2018] Lustre: Skipped 6084813 previous similar messages [Thu Jul 5 04:00:54 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 04:00:54 2018] Lustre: Skipped 6084132 previous similar messages [Thu Jul 5 04:10:54 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 04:10:54 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) Skipped 8364925 previous similar messages [Thu Jul 5 04:10:54 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 04:10:54 2018] Lustre: Skipped 5326785 previous similar messages [Thu Jul 5 04:10:54 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 04:10:54 2018] Lustre: Skipped 5326432 previous similar messages [Thu Jul 5 04:20:54 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 04:20:54 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 8464575 previous similar messages [Thu Jul 5 04:20:54 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 04:20:54 2018] Lustre: Skipped 5617440 previous similar messages [Thu Jul 5 04:20:54 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 04:20:54 2018] Lustre: Skipped 5616560 previous similar messages [Thu Jul 5 04:30:54 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 04:30:54 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 7659398 previous similar messages [Thu Jul 5 04:30:54 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 04:30:54 2018] Lustre: Skipped 5211477 previous similar messages [Thu Jul 5 04:30:54 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 04:30:54 2018] Lustre: Skipped 5210717 previous similar messages [Thu Jul 5 04:40:54 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 04:40:54 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 7441791 previous similar messages [Thu Jul 5 04:40:54 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 04:40:54 2018] Lustre: Skipped 5074172 previous similar messages [Thu Jul 5 04:40:54 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 04:40:54 2018] Lustre: Skipped 5073315 previous similar messages [Thu Jul 5 04:50:54 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 04:50:54 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 6967268 previous similar messages [Thu Jul 5 04:50:54 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 04:50:54 2018] Lustre: Skipped 4537950 previous similar messages [Thu Jul 5 04:50:54 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 04:50:54 2018] Lustre: Skipped 4537424 previous similar messages [Thu Jul 5 05:00:54 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 05:00:54 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 7356400 previous similar messages [Thu Jul 5 05:00:54 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 05:00:54 2018] Lustre: Skipped 4871575 previous similar messages [Thu Jul 5 05:00:54 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 05:00:54 2018] Lustre: Skipped 4872176 previous similar messages [Thu Jul 5 05:10:54 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 05:10:54 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 7463895 previous similar messages [Thu Jul 5 05:10:54 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 05:10:54 2018] Lustre: Skipped 5093036 previous similar messages [Thu Jul 5 05:10:54 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 05:10:54 2018] Lustre: Skipped 5092159 previous similar messages [Thu Jul 5 05:20:54 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 05:20:54 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 8568159 previous similar messages [Thu Jul 5 05:20:54 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 05:20:54 2018] Lustre: Skipped 5618118 previous similar messages [Thu Jul 5 05:20:54 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 05:20:54 2018] Lustre: Skipped 5617462 previous similar messages [Thu Jul 5 05:30:54 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 05:30:54 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 9075408 previous similar messages [Thu Jul 5 05:30:54 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 05:30:54 2018] Lustre: Skipped 5937286 previous similar messages [Thu Jul 5 05:30:54 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 05:30:54 2018] Lustre: Skipped 5936526 previous similar messages [Thu Jul 5 05:40:54 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 05:40:54 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 7518854 previous similar messages [Thu Jul 5 05:40:54 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 05:40:54 2018] Lustre: Skipped 4722257 previous similar messages [Thu Jul 5 05:40:54 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 05:40:54 2018] Lustre: Skipped 4721946 previous similar messages [Thu Jul 5 05:50:55 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 05:50:55 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 7766843 previous similar messages [Thu Jul 5 05:50:55 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 05:50:55 2018] Lustre: Skipped 5193001 previous similar messages [Thu Jul 5 05:50:55 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 05:50:55 2018] Lustre: Skipped 5192485 previous similar messages [Thu Jul 5 06:00:55 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 06:00:55 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 7745903 previous similar messages [Thu Jul 5 06:00:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 06:00:55 2018] Lustre: Skipped 5245927 previous similar messages [Thu Jul 5 06:00:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 06:00:55 2018] Lustre: Skipped 5245378 previous similar messages [Thu Jul 5 06:10:55 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 06:10:55 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 7453832 previous similar messages [Thu Jul 5 06:10:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 06:10:55 2018] Lustre: Skipped 5097768 previous similar messages [Thu Jul 5 06:10:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 06:10:55 2018] Lustre: Skipped 5097033 previous similar messages [Thu Jul 5 06:20:55 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 06:20:55 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 7459643 previous similar messages [Thu Jul 5 06:20:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 06:20:55 2018] Lustre: Skipped 5065181 previous similar messages [Thu Jul 5 06:20:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 06:20:55 2018] Lustre: Skipped 5064357 previous similar messages [Thu Jul 5 06:30:55 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 06:30:55 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 7856776 previous similar messages [Thu Jul 5 06:30:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 06:30:55 2018] Lustre: Skipped 5307812 previous similar messages [Thu Jul 5 06:30:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 06:30:55 2018] Lustre: Skipped 5307195 previous similar messages [Thu Jul 5 06:40:55 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 06:40:55 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 7077603 previous similar messages [Thu Jul 5 06:40:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 06:40:55 2018] Lustre: Skipped 4682365 previous similar messages [Thu Jul 5 06:40:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 06:40:55 2018] Lustre: Skipped 4681585 previous similar messages [Thu Jul 5 06:50:55 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 06:50:55 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 8077682 previous similar messages [Thu Jul 5 06:50:55 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 06:50:55 2018] Lustre: Skipped 5400595 previous similar messages [Thu Jul 5 06:50:55 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 06:50:55 2018] Lustre: Skipped 5399770 previous similar messages [Thu Jul 5 07:00:55 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 07:00:55 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 8320125 previous similar messages [Thu Jul 5 07:00:55 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 07:00:55 2018] Lustre: Skipped 5291223 previous similar messages [Thu Jul 5 07:00:55 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 07:00:55 2018] Lustre: Skipped 5290563 previous similar messages [Thu Jul 5 07:10:55 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 07:10:55 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 8226088 previous similar messages [Thu Jul 5 07:10:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 07:10:55 2018] Lustre: Skipped 5084795 previous similar messages [Thu Jul 5 07:10:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 07:10:55 2018] Lustre: Skipped 5084346 previous similar messages [Thu Jul 5 07:20:55 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 07:20:55 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 8649323 previous similar messages [Thu Jul 5 07:20:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 07:20:55 2018] Lustre: Skipped 5711116 previous similar messages [Thu Jul 5 07:20:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 07:20:55 2018] Lustre: Skipped 5710563 previous similar messages [Thu Jul 5 07:30:55 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 07:30:55 2018] LustreError: 4376:0:(lod_dev.c:1414:lod_sync()) Skipped 9376699 previous similar messages [Thu Jul 5 07:30:55 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 07:30:55 2018] Lustre: Skipped 6070805 previous similar messages [Thu Jul 5 07:30:55 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 07:30:55 2018] Lustre: Skipped 6070137 previous similar messages [Thu Jul 5 07:40:55 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 07:40:55 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 8305211 previous similar messages [Thu Jul 5 07:40:55 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 07:40:55 2018] Lustre: Skipped 5302821 previous similar messages [Thu Jul 5 07:40:55 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 07:40:55 2018] Lustre: Skipped 5302321 previous similar messages [Thu Jul 5 07:50:55 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 07:50:55 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 7995847 previous similar messages [Thu Jul 5 07:50:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 07:50:55 2018] Lustre: Skipped 5355127 previous similar messages [Thu Jul 5 07:50:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 07:50:55 2018] Lustre: Skipped 5354234 previous similar messages [Thu Jul 5 08:00:55 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 08:00:55 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 7552120 previous similar messages [Thu Jul 5 08:00:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 08:00:55 2018] Lustre: Skipped 4903573 previous similar messages [Thu Jul 5 08:00:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 08:00:55 2018] Lustre: Skipped 4903158 previous similar messages [Thu Jul 5 08:10:55 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 08:10:55 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 8735401 previous similar messages [Thu Jul 5 08:10:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 08:10:55 2018] Lustre: Skipped 5744993 previous similar messages [Thu Jul 5 08:10:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 08:10:55 2018] Lustre: Skipped 5744460 previous similar messages [Thu Jul 5 08:20:55 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 08:20:55 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 9183439 previous similar messages [Thu Jul 5 08:20:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 08:20:55 2018] Lustre: Skipped 5950926 previous similar messages [Thu Jul 5 08:20:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 08:20:55 2018] Lustre: Skipped 5950281 previous similar messages [Thu Jul 5 08:30:55 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 08:30:55 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 8321904 previous similar messages [Thu Jul 5 08:30:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 08:30:55 2018] Lustre: Skipped 5527506 previous similar messages [Thu Jul 5 08:30:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 08:30:55 2018] Lustre: Skipped 5527144 previous similar messages [Thu Jul 5 08:40:55 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 08:40:55 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 8326619 previous similar messages [Thu Jul 5 08:40:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 08:40:55 2018] Lustre: Skipped 5528923 previous similar messages [Thu Jul 5 08:40:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 08:40:55 2018] Lustre: Skipped 5528517 previous similar messages [Thu Jul 5 08:50:55 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 08:50:55 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 8383307 previous similar messages [Thu Jul 5 08:50:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 08:50:55 2018] Lustre: Skipped 5574627 previous similar messages [Thu Jul 5 08:50:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 08:50:55 2018] Lustre: Skipped 5574165 previous similar messages [Thu Jul 5 09:00:55 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 09:00:55 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) Skipped 8510710 previous similar messages [Thu Jul 5 09:00:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 09:00:55 2018] Lustre: Skipped 5634503 previous similar messages [Thu Jul 5 09:00:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 09:00:55 2018] Lustre: Skipped 5633999 previous similar messages [Thu Jul 5 09:10:55 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 09:10:55 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 7582854 previous similar messages [Thu Jul 5 09:10:55 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 09:10:55 2018] Lustre: Skipped 5152666 previous similar messages [Thu Jul 5 09:10:55 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 09:10:55 2018] Lustre: Skipped 5151780 previous similar messages [Thu Jul 5 09:20:55 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 09:20:55 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 8337761 previous similar messages [Thu Jul 5 09:20:55 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 09:20:55 2018] Lustre: Skipped 5526991 previous similar messages [Thu Jul 5 09:20:55 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 09:20:55 2018] Lustre: Skipped 5525975 previous similar messages [Thu Jul 5 09:30:55 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 09:30:55 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 8371765 previous similar messages [Thu Jul 5 09:30:56 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 09:30:56 2018] Lustre: Skipped 5556049 previous similar messages [Thu Jul 5 09:30:56 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 09:30:56 2018] Lustre: Skipped 5555394 previous similar messages [Thu Jul 5 09:40:56 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 09:40:56 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 8114322 previous similar messages [Thu Jul 5 09:40:56 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 09:40:56 2018] Lustre: Skipped 5435463 previous similar messages [Thu Jul 5 09:40:56 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 09:40:56 2018] Lustre: Skipped 5434961 previous similar messages [Thu Jul 5 09:50:56 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 09:50:56 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 7625731 previous similar messages [Thu Jul 5 09:50:56 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 09:50:56 2018] Lustre: Skipped 5166208 previous similar messages [Thu Jul 5 09:50:56 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 09:50:56 2018] Lustre: Skipped 5165401 previous similar messages [Thu Jul 5 10:00:56 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 10:00:56 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 8691726 previous similar messages [Thu Jul 5 10:00:56 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 10:00:56 2018] Lustre: Skipped 5732320 previous similar messages [Thu Jul 5 10:00:56 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 10:00:56 2018] Lustre: Skipped 5731234 previous similar messages [Thu Jul 5 10:10:56 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 10:10:56 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) Skipped 9777458 previous similar messages [Thu Jul 5 10:10:56 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 10:10:56 2018] Lustre: Skipped 7083326 previous similar messages [Thu Jul 5 10:10:56 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 10:10:56 2018] Lustre: Skipped 7083186 previous similar messages [Thu Jul 5 10:20:56 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 10:20:56 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) Skipped 12959165 previous similar messages [Thu Jul 5 10:20:56 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 10:20:56 2018] Lustre: Skipped 10709203 previous similar messages [Thu Jul 5 10:20:56 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 10:20:56 2018] Lustre: Skipped 10709152 previous similar messages [Thu Jul 5 10:30:56 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 10:30:56 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 14450401 previous similar messages [Thu Jul 5 10:30:56 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 10:30:56 2018] Lustre: Skipped 12179679 previous similar messages [Thu Jul 5 10:30:56 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 10:30:56 2018] Lustre: Skipped 12179591 previous similar messages [Thu Jul 5 10:40:56 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 10:40:56 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) Skipped 14636630 previous similar messages [Thu Jul 5 10:40:56 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 10:40:56 2018] Lustre: Skipped 12418651 previous similar messages [Thu Jul 5 10:40:56 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 10:40:56 2018] Lustre: Skipped 12418498 previous similar messages [Thu Jul 5 10:50:56 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 10:50:56 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 14306294 previous similar messages [Thu Jul 5 10:50:56 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 10:50:56 2018] Lustre: Skipped 12026007 previous similar messages [Thu Jul 5 10:50:56 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 10:50:56 2018] Lustre: Skipped 12025900 previous similar messages [Thu Jul 5 11:00:56 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 11:00:56 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 14117809 previous similar messages [Thu Jul 5 11:00:56 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 11:00:56 2018] Lustre: Skipped 11717947 previous similar messages [Thu Jul 5 11:00:56 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 11:00:56 2018] Lustre: Skipped 11717739 previous similar messages [Thu Jul 5 11:10:56 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 11:10:56 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) Skipped 14954047 previous similar messages [Thu Jul 5 11:10:56 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 11:10:56 2018] Lustre: Skipped 12193421 previous similar messages [Thu Jul 5 11:10:56 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 11:10:56 2018] Lustre: Skipped 12193360 previous similar messages [Thu Jul 5 11:20:56 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 11:20:56 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 17431202 previous similar messages [Thu Jul 5 11:20:56 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 11:20:56 2018] Lustre: Skipped 12437754 previous similar messages [Thu Jul 5 11:20:56 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 11:20:56 2018] Lustre: Skipped 12437738 previous similar messages [Thu Jul 5 11:30:56 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 11:30:56 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) Skipped 17561842 previous similar messages [Thu Jul 5 11:30:56 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 11:30:56 2018] Lustre: Skipped 12060666 previous similar messages [Thu Jul 5 11:30:56 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 11:30:56 2018] Lustre: Skipped 12060594 previous similar messages [Thu Jul 5 11:40:56 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 11:40:56 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) Skipped 16811087 previous similar messages [Thu Jul 5 11:40:56 2018] Lustre: lustre-MDT0000: Client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) reconnecting [Thu Jul 5 11:40:56 2018] Lustre: Skipped 11627827 previous similar messages [Thu Jul 5 11:40:56 2018] Lustre: lustre-MDT0000: Connection restored to c1fdfd64-45d9-1da4-3d64-51cc639eea76 (at 172.16.230.55@o2ib) [Thu Jul 5 11:40:56 2018] Lustre: Skipped 11627700 previous similar messages [Thu Jul 5 11:50:56 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 11:50:56 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 16561108 previous similar messages [Thu Jul 5 11:50:56 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 11:50:56 2018] Lustre: Skipped 11414829 previous similar messages [Thu Jul 5 11:50:56 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 11:50:56 2018] Lustre: Skipped 11414603 previous similar messages [Thu Jul 5 12:00:56 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 12:00:56 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) Skipped 17223299 previous similar messages [Thu Jul 5 12:00:56 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 12:00:56 2018] Lustre: Skipped 11591720 previous similar messages [Thu Jul 5 12:00:56 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 12:00:56 2018] Lustre: Skipped 11591542 previous similar messages [Thu Jul 5 12:10:56 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 12:10:56 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 11374127 previous similar messages [Thu Jul 5 12:10:56 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 12:10:56 2018] Lustre: Skipped 8896653 previous similar messages [Thu Jul 5 12:10:56 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 12:10:56 2018] Lustre: Skipped 8896542 previous similar messages [Thu Jul 5 12:20:56 2018] LustreError: 6412:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 12:20:56 2018] LustreError: 6412:0:(lod_dev.c:1414:lod_sync()) Skipped 15132108 previous similar messages [Thu Jul 5 12:20:56 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 12:20:56 2018] Lustre: Skipped 12333047 previous similar messages [Thu Jul 5 12:20:56 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 12:20:56 2018] Lustre: Skipped 12333052 previous similar messages [Thu Jul 5 12:30:56 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 12:30:56 2018] LustreError: 39620:0:(lod_dev.c:1414:lod_sync()) Skipped 15082426 previous similar messages [Thu Jul 5 12:30:56 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 12:30:56 2018] Lustre: Skipped 12324476 previous similar messages [Thu Jul 5 12:30:56 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 12:30:56 2018] Lustre: Skipped 12324533 previous similar messages [Thu Jul 5 12:40:56 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 12:40:56 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 15023617 previous similar messages [Thu Jul 5 12:40:56 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 12:40:56 2018] Lustre: Skipped 12267148 previous similar messages [Thu Jul 5 12:40:56 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 12:40:56 2018] Lustre: Skipped 12267074 previous similar messages [Thu Jul 5 12:50:56 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 12:50:56 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) Skipped 14807591 previous similar messages [Thu Jul 5 12:50:56 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 12:50:56 2018] Lustre: Skipped 12041518 previous similar messages [Thu Jul 5 12:50:56 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 12:50:56 2018] Lustre: Skipped 12041540 previous similar messages [Thu Jul 5 12:52:15 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe316c50 x1604331598127600/t0(0) o37->74dc27c9-7eb7-6236-850f-6507aace669b@172.16.230.55@o2ib:192/0 lens 568/440 e 0 to 0 dl 1530758917 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 12:52:15 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880cf50e6600 x1604331598127600/t0(0) o37->74dc27c9-7eb7-6236-850f-6507aace669b@172.16.230.55@o2ib:192/0 lens 568/440 e 0 to 0 dl 1530758917 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 12:52:15 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Thu Jul 5 12:52:21 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880504b0b300 x1604331598498912/t0(0) o37->74dc27c9-7eb7-6236-850f-6507aace669b@172.16.230.55@o2ib:197/0 lens 568/440 e 0 to 0 dl 1530758922 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 12:52:23 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8803f37b1500 x1604331598693648/t0(0) o37->74dc27c9-7eb7-6236-850f-6507aace669b@172.16.230.55@o2ib:200/0 lens 568/440 e 0 to 0 dl 1530758925 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 12:52:23 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880c044a2400 x1604331598693648/t0(0) o37->74dc27c9-7eb7-6236-850f-6507aace669b@172.16.230.55@o2ib:200/0 lens 568/440 e 0 to 0 dl 1530758925 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 12:52:23 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.230.55@o2ib [Thu Jul 5 12:52:23 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 4 previous similar messages [Thu Jul 5 12:52:23 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 11 previous similar messages [Thu Jul 5 12:55:12 2018] LNetError: 2851:0:(o2iblnd_cb.c:3251:kiblnd_check_txs_locked()) Timed out tx: active_txs, 12 seconds [Thu Jul 5 12:55:12 2018] LNetError: 2851:0:(o2iblnd_cb.c:3326:kiblnd_check_conns()) Timed out RDMA with 172.16.230.55@o2ib (167): c: 7, oc: 0, rc: 8 [Thu Jul 5 12:55:47 2018] Lustre: MGS: haven't heard from client f906dc98-eb62-a3c0-ce27-644ca41101d2 (at 172.16.230.55@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f43aed000, cur 1530759123 expire 1530758973 last 1530758896 [Thu Jul 5 12:56:11 2018] Lustre: lustre-MDT0000: haven't heard from client 74dc27c9-7eb7-6236-850f-6507aace669b (at 172.16.230.55@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817b976a000, cur 1530759146 expire 1530758996 last 1530758919 [Thu Jul 5 13:00:56 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 13:00:56 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) Skipped 11393906 previous similar messages [Thu Jul 5 13:00:56 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 13:00:56 2018] Lustre: Skipped 9156053 previous similar messages [Thu Jul 5 13:00:56 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 13:00:56 2018] Lustre: Skipped 9155291 previous similar messages [Thu Jul 5 13:10:56 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 13:10:56 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 12876880 previous similar messages [Thu Jul 5 13:10:57 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 13:10:57 2018] Lustre: Skipped 9919525 previous similar messages [Thu Jul 5 13:10:57 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 13:10:57 2018] Lustre: Skipped 9919353 previous similar messages [Thu Jul 5 13:20:57 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 13:20:57 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 12974668 previous similar messages [Thu Jul 5 13:20:57 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 13:20:57 2018] Lustre: Skipped 9904588 previous similar messages [Thu Jul 5 13:20:57 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 13:20:57 2018] Lustre: Skipped 9904544 previous similar messages [Thu Jul 5 13:30:57 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 13:30:57 2018] LustreError: 5082:0:(lod_dev.c:1414:lod_sync()) Skipped 11556187 previous similar messages [Thu Jul 5 13:30:57 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 13:30:57 2018] Lustre: Skipped 8774852 previous similar messages [Thu Jul 5 13:30:57 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 13:30:57 2018] Lustre: Skipped 8774457 previous similar messages [Thu Jul 5 13:40:57 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 13:40:57 2018] LustreError: 5564:0:(lod_dev.c:1414:lod_sync()) Skipped 12130649 previous similar messages [Thu Jul 5 13:40:57 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 13:40:57 2018] Lustre: Skipped 9456942 previous similar messages [Thu Jul 5 13:40:57 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 13:40:57 2018] Lustre: Skipped 9456352 previous similar messages [Thu Jul 5 13:50:57 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 13:50:57 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) Skipped 10030488 previous similar messages [Thu Jul 5 13:50:57 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 13:50:57 2018] Lustre: Skipped 7813603 previous similar messages [Thu Jul 5 13:50:57 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 13:50:57 2018] Lustre: Skipped 7813181 previous similar messages [Thu Jul 5 14:00:57 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 14:00:57 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) Skipped 13007152 previous similar messages [Thu Jul 5 14:00:57 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 14:00:57 2018] Lustre: Skipped 9959262 previous similar messages [Thu Jul 5 14:00:57 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 14:00:57 2018] Lustre: Skipped 9959191 previous similar messages [Thu Jul 5 14:10:57 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 14:10:57 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) Skipped 12867553 previous similar messages [Thu Jul 5 14:10:57 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 14:10:57 2018] Lustre: Skipped 9909801 previous similar messages [Thu Jul 5 14:10:57 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 14:10:57 2018] Lustre: Skipped 9909592 previous similar messages [Thu Jul 5 14:20:57 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 14:20:57 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) Skipped 12011558 previous similar messages [Thu Jul 5 14:20:57 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 14:20:57 2018] Lustre: Skipped 9492868 previous similar messages [Thu Jul 5 14:20:57 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 14:20:57 2018] Lustre: Skipped 9492842 previous similar messages [Thu Jul 5 14:30:57 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 14:30:57 2018] LustreError: 4378:0:(lod_dev.c:1414:lod_sync()) Skipped 12414750 previous similar messages [Thu Jul 5 14:30:57 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 14:30:57 2018] Lustre: Skipped 9702354 previous similar messages [Thu Jul 5 14:30:57 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 14:30:57 2018] Lustre: Skipped 9701845 previous similar messages [Thu Jul 5 14:40:57 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 14:40:57 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) Skipped 12584013 previous similar messages [Thu Jul 5 14:40:57 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 14:40:57 2018] Lustre: Skipped 10100280 previous similar messages [Thu Jul 5 14:40:57 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 14:40:57 2018] Lustre: Skipped 10100083 previous similar messages [Thu Jul 5 14:50:57 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 14:50:57 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 12749594 previous similar messages [Thu Jul 5 14:50:57 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 14:50:57 2018] Lustre: Skipped 9745426 previous similar messages [Thu Jul 5 14:50:57 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 14:50:57 2018] Lustre: Skipped 9745361 previous similar messages [Thu Jul 5 15:00:57 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 15:00:57 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 12474409 previous similar messages [Thu Jul 5 15:00:57 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 15:00:57 2018] Lustre: Skipped 9555301 previous similar messages [Thu Jul 5 15:00:57 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 15:00:57 2018] Lustre: Skipped 9555101 previous similar messages [Thu Jul 5 15:03:37 2018] Lustre: lustre-MDT0000: haven't heard from client f4a00223-0c54-b2cb-8b2f-551ebd3bc19f (at 172.16.230.91@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b05681000, cur 1530766792 expire 1530766642 last 1530766565 [Thu Jul 5 15:10:57 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 15:10:57 2018] LustreError: 6221:0:(lod_dev.c:1414:lod_sync()) Skipped 11953588 previous similar messages [Thu Jul 5 15:10:57 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 15:10:57 2018] Lustre: Skipped 9508584 previous similar messages [Thu Jul 5 15:10:57 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 15:10:57 2018] Lustre: Skipped 9508563 previous similar messages [Thu Jul 5 15:20:57 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 15:20:57 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 12102925 previous similar messages [Thu Jul 5 15:20:57 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 15:20:57 2018] Lustre: Skipped 9739752 previous similar messages [Thu Jul 5 15:20:57 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 15:20:57 2018] Lustre: Skipped 9739327 previous similar messages [Thu Jul 5 15:30:57 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 15:30:57 2018] LustreError: 4380:0:(lod_dev.c:1414:lod_sync()) Skipped 13592680 previous similar messages [Thu Jul 5 15:30:57 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 15:30:57 2018] Lustre: Skipped 10432677 previous similar messages [Thu Jul 5 15:30:57 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 15:30:57 2018] Lustre: Skipped 10432634 previous similar messages [Thu Jul 5 15:40:57 2018] LustreError: 6412:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 15:40:57 2018] LustreError: 6412:0:(lod_dev.c:1414:lod_sync()) Skipped 15885315 previous similar messages [Thu Jul 5 15:40:57 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 15:40:57 2018] Lustre: Skipped 12859825 previous similar messages [Thu Jul 5 15:40:57 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 15:40:57 2018] Lustre: Skipped 12861009 previous similar messages [Thu Jul 5 15:50:57 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 15:50:57 2018] LustreError: 25630:0:(lod_dev.c:1414:lod_sync()) Skipped 15406525 previous similar messages [Thu Jul 5 15:50:57 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 15:50:57 2018] Lustre: Skipped 12384508 previous similar messages [Thu Jul 5 15:50:57 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 15:50:57 2018] Lustre: Skipped 12383498 previous similar messages [Thu Jul 5 16:00:57 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 16:00:57 2018] LustreError: 146442:0:(lod_dev.c:1414:lod_sync()) Skipped 15463355 previous similar messages [Thu Jul 5 16:00:57 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 16:00:57 2018] Lustre: Skipped 12393285 previous similar messages [Thu Jul 5 16:00:57 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 16:00:57 2018] Lustre: Skipped 12392328 previous similar messages [Thu Jul 5 16:05:38 2018] Lustre: lustre-MDT0000: haven't heard from client 3bbd4580-e1cc-4dfb-ce75-65aba5f2a8cf (at 172.16.230.91@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202a270800, cur 1530770513 expire 1530770363 last 1530770286 [Thu Jul 5 16:10:57 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 16:10:57 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 14486260 previous similar messages [Thu Jul 5 16:10:57 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 16:10:57 2018] Lustre: Skipped 11490768 previous similar messages [Thu Jul 5 16:10:57 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 16:10:57 2018] Lustre: Skipped 11490093 previous similar messages [Thu Jul 5 16:20:57 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 16:20:57 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) Skipped 13094368 previous similar messages [Thu Jul 5 16:20:57 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 16:20:57 2018] Lustre: Skipped 10542524 previous similar messages [Thu Jul 5 16:20:57 2018] Lustre: lustre-MDT0000: Client cc557fcf-0ca7-eaed-bd9d-414250ebeb4a (at 172.16.230.91@o2ib) reconnecting [Thu Jul 5 16:20:57 2018] Lustre: Skipped 10543500 previous similar messages [Thu Jul 5 16:30:57 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 16:30:57 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) Skipped 14750569 previous similar messages [Thu Jul 5 16:30:57 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 16:30:57 2018] Lustre: Skipped 11944533 previous similar messages [Thu Jul 5 16:30:57 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 16:30:57 2018] Lustre: Skipped 11943139 previous similar messages [Thu Jul 5 16:40:57 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 16:40:57 2018] LustreError: 4896:0:(lod_dev.c:1414:lod_sync()) Skipped 15546788 previous similar messages [Thu Jul 5 16:40:57 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 16:40:57 2018] Lustre: Skipped 12554988 previous similar messages [Thu Jul 5 16:40:57 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 16:40:57 2018] Lustre: Skipped 12553680 previous similar messages [Thu Jul 5 16:46:22 2018] Lustre: lustre-MDT0000: haven't heard from client cc557fcf-0ca7-eaed-bd9d-414250ebeb4a (at 172.16.230.91@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ddd604000, cur 1530772957 expire 1530772807 last 1530772730 [Thu Jul 5 16:50:57 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 16:50:57 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) Skipped 13476884 previous similar messages [Thu Jul 5 16:50:57 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 16:50:57 2018] Lustre: Skipped 10550924 previous similar messages [Thu Jul 5 16:50:57 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 16:50:57 2018] Lustre: Skipped 10550558 previous similar messages [Thu Jul 5 17:00:58 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 17:00:58 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 13167968 previous similar messages [Thu Jul 5 17:00:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 17:00:58 2018] Lustre: Skipped 10065832 previous similar messages [Thu Jul 5 17:00:58 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 17:00:58 2018] Lustre: Skipped 10065748 previous similar messages [Thu Jul 5 17:10:58 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 17:10:58 2018] LustreError: 5696:0:(lod_dev.c:1414:lod_sync()) Skipped 12212942 previous similar messages [Thu Jul 5 17:10:58 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 17:10:58 2018] Lustre: Skipped 9638162 previous similar messages [Thu Jul 5 17:10:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 17:10:58 2018] Lustre: Skipped 9638238 previous similar messages [Thu Jul 5 17:20:58 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 17:20:58 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 12786693 previous similar messages [Thu Jul 5 17:20:58 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 17:20:58 2018] Lustre: Skipped 9961728 previous similar messages [Thu Jul 5 17:20:58 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 17:20:58 2018] Lustre: Skipped 9961191 previous similar messages [Thu Jul 5 17:30:58 2018] LustreError: 146455:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 17:30:58 2018] LustreError: 146455:0:(lod_dev.c:1414:lod_sync()) Skipped 12011081 previous similar messages [Thu Jul 5 17:30:58 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 17:30:58 2018] Lustre: Skipped 9138159 previous similar messages [Thu Jul 5 17:30:58 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 17:30:58 2018] Lustre: Skipped 9137810 previous similar messages [Thu Jul 5 17:40:58 2018] LustreError: 146455:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 17:40:58 2018] LustreError: 146455:0:(lod_dev.c:1414:lod_sync()) Skipped 12852774 previous similar messages [Thu Jul 5 17:40:58 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 17:40:58 2018] Lustre: Skipped 9997003 previous similar messages [Thu Jul 5 17:40:58 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 17:40:58 2018] Lustre: Skipped 9996625 previous similar messages [Thu Jul 5 17:50:58 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 17:50:58 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) Skipped 12639706 previous similar messages [Thu Jul 5 17:50:58 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 17:50:58 2018] Lustre: Skipped 9896835 previous similar messages [Thu Jul 5 17:50:58 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 17:50:58 2018] Lustre: Skipped 9896266 previous similar messages [Thu Jul 5 18:00:58 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 18:00:58 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) Skipped 12838632 previous similar messages [Thu Jul 5 18:00:58 2018] Lustre: lustre-MDT0000: Client 5d217489-c48d-eddd-0cfc-4b27a6b417c8 (at 172.16.229.37@o2ib) reconnecting [Thu Jul 5 18:00:58 2018] Lustre: Skipped 9935398 previous similar messages [Thu Jul 5 18:00:58 2018] Lustre: lustre-MDT0000: Connection restored to e595ded6-0f62-2417-22fa-11943a3478ba (at 172.16.229.37@o2ib) [Thu Jul 5 18:00:58 2018] Lustre: Skipped 9935016 previous similar messages [Thu Jul 5 18:10:58 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 18:10:58 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) Skipped 12387648 previous similar messages [Thu Jul 5 18:10:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 18:10:58 2018] Lustre: Skipped 9844890 previous similar messages [Thu Jul 5 18:10:58 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 18:10:58 2018] Lustre: Skipped 9844222 previous similar messages [Thu Jul 5 18:20:58 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 18:20:58 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) Skipped 14507449 previous similar messages [Thu Jul 5 18:20:58 2018] Lustre: lustre-MDT0000: Connection restored to 8ac51542-2078-f807-3e02-dcdc5ad157c3 (at 172.16.230.91@o2ib) [Thu Jul 5 18:20:58 2018] Lustre: Skipped 12340292 previous similar messages [Thu Jul 5 18:20:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 18:20:58 2018] Lustre: Skipped 12342222 previous similar messages [Thu Jul 5 18:30:58 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 18:30:58 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 13959269 previous similar messages [Thu Jul 5 18:30:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 18:30:58 2018] Lustre: Skipped 11675864 previous similar messages [Thu Jul 5 18:30:58 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 18:30:58 2018] Lustre: Skipped 11674105 previous similar messages [Thu Jul 5 18:31:03 2018] LustreError: 5022:0:(ldlm_lockd.c:690:ldlm_handle_ast_error()) ### client (nid 172.16.229.37@o2ib) returned error from blocking AST (req@ffff8802eae0ce00 x1601781845684704 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: ffff88190971f740/0x88fac5e9b04928f lrc: 4/0,0 mode: PR/PR res: [0x200009cb9:0xaf48:0x0].0x0 bits 0x13/0x0 rrc: 10 type: IBT flags: 0x60200400000020 nid: 172.16.229.37@o2ib remote: 0xa79570bc6ee3427b expref: 1793 pid: 5564 timeout: 3201734 lvb_type: 0 [Thu Jul 5 18:31:03 2018] LustreError: 138-a: lustre-MDT0000: A client on nid 172.16.229.37@o2ib was evicted due to a lock blocking callback time out: rc -107 [Thu Jul 5 18:32:12 2018] Lustre: lustre-MDT0000: haven't heard from client 0bf6c8a9-e91b-c565-d007-4fff42c51503 (at 172.16.230.91@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882017f3bc00, cur 1530779306 expire 1530779156 last 1530779079 [Thu Jul 5 18:40:58 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 18:40:58 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 8949045 previous similar messages [Thu Jul 5 18:40:58 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 18:40:58 2018] Lustre: Skipped 6531073 previous similar messages [Thu Jul 5 18:40:58 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 18:40:58 2018] Lustre: Skipped 6531060 previous similar messages [Thu Jul 5 18:50:58 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 18:50:58 2018] LustreError: 156579:0:(lod_dev.c:1414:lod_sync()) Skipped 8598285 previous similar messages [Thu Jul 5 18:50:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 18:50:58 2018] Lustre: Skipped 6367830 previous similar messages [Thu Jul 5 18:50:58 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 18:50:58 2018] Lustre: Skipped 6367826 previous similar messages [Thu Jul 5 19:00:58 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 19:00:58 2018] LustreError: 39619:0:(lod_dev.c:1414:lod_sync()) Skipped 8653856 previous similar messages [Thu Jul 5 19:00:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 19:00:58 2018] Lustre: Skipped 6397145 previous similar messages [Thu Jul 5 19:00:58 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 19:00:58 2018] Lustre: Skipped 6397135 previous similar messages [Thu Jul 5 19:10:58 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 19:10:58 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 8935875 previous similar messages [Thu Jul 5 19:10:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 19:10:58 2018] Lustre: Skipped 6567275 previous similar messages [Thu Jul 5 19:10:58 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 19:10:58 2018] Lustre: Skipped 6567260 previous similar messages [Thu Jul 5 19:20:58 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 19:20:58 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 8612561 previous similar messages [Thu Jul 5 19:20:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 19:20:58 2018] Lustre: Skipped 6253497 previous similar messages [Thu Jul 5 19:20:58 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 19:20:58 2018] Lustre: Skipped 6253480 previous similar messages [Thu Jul 5 19:30:58 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 19:30:58 2018] LustreError: 39617:0:(lod_dev.c:1414:lod_sync()) Skipped 8116867 previous similar messages [Thu Jul 5 19:30:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 19:30:58 2018] Lustre: Skipped 5950354 previous similar messages [Thu Jul 5 19:30:58 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 19:30:58 2018] Lustre: Skipped 5950342 previous similar messages [Thu Jul 5 19:40:58 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 19:40:58 2018] LustreError: 48140:0:(lod_dev.c:1414:lod_sync()) Skipped 8706766 previous similar messages [Thu Jul 5 19:40:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 19:40:58 2018] Lustre: Skipped 6290309 previous similar messages [Thu Jul 5 19:40:58 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 19:40:58 2018] Lustre: Skipped 6290293 previous similar messages [Thu Jul 5 19:50:58 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 19:50:58 2018] LustreError: 39618:0:(lod_dev.c:1414:lod_sync()) Skipped 8335450 previous similar messages [Thu Jul 5 19:50:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 19:50:58 2018] Lustre: Skipped 6064798 previous similar messages [Thu Jul 5 19:50:58 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 19:50:58 2018] Lustre: Skipped 6064788 previous similar messages [Thu Jul 5 20:00:58 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 20:00:58 2018] LustreError: 39616:0:(lod_dev.c:1414:lod_sync()) Skipped 8325442 previous similar messages [Thu Jul 5 20:00:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 20:00:58 2018] Lustre: Skipped 5930844 previous similar messages [Thu Jul 5 20:00:58 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 20:00:58 2018] Lustre: Skipped 5930837 previous similar messages [Thu Jul 5 20:10:58 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 20:10:58 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) Skipped 8249663 previous similar messages [Thu Jul 5 20:10:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 20:10:58 2018] Lustre: Skipped 5992395 previous similar messages [Thu Jul 5 20:10:58 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 20:10:58 2018] Lustre: Skipped 5992387 previous similar messages [Thu Jul 5 20:20:58 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 20:20:58 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 7161248 previous similar messages [Thu Jul 5 20:20:58 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 20:20:58 2018] Lustre: Skipped 4746494 previous similar messages [Thu Jul 5 20:20:58 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 20:20:58 2018] Lustre: Skipped 4746481 previous similar messages [Thu Jul 5 20:30:58 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 20:30:58 2018] LustreError: 4377:0:(lod_dev.c:1414:lod_sync()) Skipped 8473006 previous similar messages [Thu Jul 5 20:30:58 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 20:30:58 2018] Lustre: Skipped 5597590 previous similar messages [Thu Jul 5 20:30:58 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 20:30:58 2018] Lustre: Skipped 5597574 previous similar messages [Thu Jul 5 20:40:58 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 20:40:58 2018] LustreError: 103699:0:(lod_dev.c:1414:lod_sync()) Skipped 8350439 previous similar messages [Thu Jul 5 20:40:59 2018] Lustre: lustre-MDT0000: Client ee0c897a-a8d1-8425-6023-ab3f293a7d36 (at 172.16.229.45@o2ib) reconnecting [Thu Jul 5 20:40:59 2018] Lustre: Skipped 5635297 previous similar messages [Thu Jul 5 20:40:59 2018] Lustre: lustre-MDT0000: Connection restored to d4cae4db-fe57-ac3c-88b9-f24051c987d9 (at 172.16.229.45@o2ib) [Thu Jul 5 20:40:59 2018] Lustre: Skipped 5635294 previous similar messages [Thu Jul 5 20:50:59 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 20:50:59 2018] LustreError: 89766:0:(lod_dev.c:1414:lod_sync()) Skipped 6329954 previous similar messages [Thu Jul 5 20:50:59 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 20:50:59 2018] Lustre: Skipped 3546051 previous similar messages [Thu Jul 5 20:50:59 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 20:50:59 2018] Lustre: Skipped 3546049 previous similar messages [Thu Jul 5 21:00:59 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 21:00:59 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 5407004 previous similar messages [Thu Jul 5 21:00:59 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 21:00:59 2018] Lustre: Skipped 2703755 previous similar messages [Thu Jul 5 21:00:59 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 21:00:59 2018] Lustre: Skipped 2703755 previous similar messages [Thu Jul 5 21:10:59 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 21:10:59 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 5863522 previous similar messages [Thu Jul 5 21:10:59 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 21:10:59 2018] Lustre: Skipped 2932009 previous similar messages [Thu Jul 5 21:10:59 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 21:10:59 2018] Lustre: Skipped 2932009 previous similar messages [Thu Jul 5 21:20:59 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 21:20:59 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 4487618 previous similar messages [Thu Jul 5 21:20:59 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 21:20:59 2018] Lustre: Skipped 2244043 previous similar messages [Thu Jul 5 21:20:59 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 21:20:59 2018] Lustre: Skipped 2244043 previous similar messages [Thu Jul 5 21:30:59 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 21:30:59 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 5281083 previous similar messages [Thu Jul 5 21:30:59 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 21:30:59 2018] Lustre: Skipped 2641042 previous similar messages [Thu Jul 5 21:30:59 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 21:30:59 2018] Lustre: Skipped 2641042 previous similar messages [Thu Jul 5 21:40:59 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 21:40:59 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 5225057 previous similar messages [Thu Jul 5 21:40:59 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 21:40:59 2018] Lustre: Skipped 2612759 previous similar messages [Thu Jul 5 21:40:59 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 21:40:59 2018] Lustre: Skipped 2612759 previous similar messages [Thu Jul 5 21:50:59 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 21:50:59 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 4698729 previous similar messages [Thu Jul 5 21:50:59 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 21:50:59 2018] Lustre: Skipped 2349556 previous similar messages [Thu Jul 5 21:50:59 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 21:50:59 2018] Lustre: Skipped 2349556 previous similar messages [Thu Jul 5 22:00:59 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 22:00:59 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 4721841 previous similar messages [Thu Jul 5 22:00:59 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 22:00:59 2018] Lustre: Skipped 2360988 previous similar messages [Thu Jul 5 22:00:59 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 22:00:59 2018] Lustre: Skipped 2360988 previous similar messages [Thu Jul 5 22:10:59 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 22:10:59 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) Skipped 4729161 previous similar messages [Thu Jul 5 22:10:59 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 22:10:59 2018] Lustre: Skipped 2364720 previous similar messages [Thu Jul 5 22:10:59 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 22:10:59 2018] Lustre: Skipped 2364720 previous similar messages [Thu Jul 5 22:20:59 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 22:20:59 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 22:20:59 2018] LustreError: 99713:0:(lod_dev.c:1414:lod_sync()) Skipped 4505904 previous similar messages [Thu Jul 5 22:20:59 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 22:20:59 2018] Lustre: Skipped 2253047 previous similar messages [Thu Jul 5 22:20:59 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 22:20:59 2018] Lustre: Skipped 2253047 previous similar messages [Thu Jul 5 22:30:59 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 22:30:59 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 4554516 previous similar messages [Thu Jul 5 22:30:59 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 22:30:59 2018] Lustre: Skipped 2277337 previous similar messages [Thu Jul 5 22:30:59 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 22:30:59 2018] Lustre: Skipped 2277337 previous similar messages [Thu Jul 5 22:40:59 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 22:40:59 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 4630813 previous similar messages [Thu Jul 5 22:40:59 2018] Lustre: lustre-MDT0000: Client 4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad (at 172.16.229.39@o2ib) reconnecting [Thu Jul 5 22:40:59 2018] Lustre: Skipped 2682373 previous similar messages [Thu Jul 5 22:40:59 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Thu Jul 5 22:40:59 2018] Lustre: Skipped 2682240 previous similar messages [Thu Jul 5 22:41:03 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8807dcfa6300 x1604934179130176/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:32/0 lens 568/440 e 0 to 0 dl 1530794242 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 22:41:03 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 10 previous similar messages [Thu Jul 5 22:41:11 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe312850 x1604934180070384/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:40/0 lens 568/440 e 0 to 0 dl 1530794250 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 22:41:11 2018] LustreError: 161086:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8804b169a100 x1604934180070384/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:40/0 lens 568/440 e 0 to 0 dl 1530794250 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 22:41:11 2018] LustreError: 161086:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 1 previous similar message [Thu Jul 5 22:41:11 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Thu Jul 5 22:41:12 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8804b169a700 x1604934180070384/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:40/0 lens 568/440 e 0 to 0 dl 1530794250 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 22:41:12 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 1 previous similar message [Thu Jul 5 22:41:14 2018] LustreError: 161086:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880c403fb300 x1604934180392592/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:43/0 lens 568/440 e 0 to 0 dl 1530794253 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 22:41:14 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffe315050 x1604934180392592/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:43/0 lens 568/440 e 0 to 0 dl 1530794253 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 22:41:14 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 1 previous similar message [Thu Jul 5 22:41:14 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Thu Jul 5 22:41:14 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 1 previous similar message [Thu Jul 5 22:41:14 2018] LustreError: 161086:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Thu Jul 5 22:41:17 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880923318300 x1604934180722800/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:83/0 lens 568/440 e 0 to 0 dl 1530794293 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 22:41:17 2018] LustreError: 161086:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8809e9330300 x1604934180722800/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:83/0 lens 568/440 e 0 to 0 dl 1530794293 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 22:41:17 2018] LustreError: 161086:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 1 previous similar message [Thu Jul 5 22:41:17 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Thu Jul 5 22:41:22 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8809163de600 x1604934181160560/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:87/0 lens 568/440 e 0 to 0 dl 1530794297 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 22:41:22 2018] LustreError: 161089:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 2 previous similar messages [Thu Jul 5 22:41:26 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2fb850 x1604934181701120/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:92/0 lens 568/440 e 0 to 0 dl 1530794302 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 22:41:26 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Thu Jul 5 22:41:26 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 1 previous similar message [Thu Jul 5 22:41:32 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88049732b600 x1604934182364384/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:98/0 lens 568/440 e 0 to 0 dl 1530794308 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 22:41:32 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 2 previous similar messages [Thu Jul 5 22:41:37 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Thu Jul 5 22:41:38 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8802e04e9500 x1604934183016416/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:104/0 lens 568/440 e 0 to 0 dl 1530794314 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 22:41:38 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Thu Jul 5 22:41:52 2018] LustreError: 161086:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880af9e1b000 x1604934184644336/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:118/0 lens 568/440 e 0 to 0 dl 1530794328 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 22:41:52 2018] LustreError: 161086:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 11 previous similar messages [Thu Jul 5 22:41:58 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8808433e9b00 x1604934185316944/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:124/0 lens 568/440 e 0 to 0 dl 1530794334 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 22:41:58 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Thu Jul 5 22:41:58 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 5 previous similar messages [Thu Jul 5 22:41:58 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 3 previous similar messages [Thu Jul 5 22:43:15 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8807af058900 x1604934193994336/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:201/0 lens 568/440 e 0 to 0 dl 1530794411 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 22:43:15 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8807f3a88c00 x1604934193994336/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:201/0 lens 568/440 e 0 to 0 dl 1530794411 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 22:43:15 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 9 previous similar messages [Thu Jul 5 22:43:15 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Thu Jul 5 22:43:15 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 3 previous similar messages [Thu Jul 5 22:43:15 2018] LustreError: 161089:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 2 previous similar messages [Thu Jul 5 22:44:36 2018] LustreError: 161086:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88064ae19800 x1604934203089232/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:282/0 lens 568/440 e 0 to 0 dl 1530794492 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 22:44:36 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Thu Jul 5 22:44:36 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 8 previous similar messages [Thu Jul 5 22:44:36 2018] LustreError: 161086:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 19 previous similar messages [Thu Jul 5 22:44:37 2018] LustreError: 161086:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe313c50 x1604934203199168/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:283/0 lens 568/440 e 0 to 0 dl 1530794493 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 22:44:37 2018] LustreError: 161086:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 7 previous similar messages [Thu Jul 5 22:46:49 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2fb050 x1604934217775056/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:414/0 lens 568/440 e 0 to 0 dl 1530794624 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 22:46:49 2018] LustreError: 4382:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 20 previous similar messages [Thu Jul 5 22:47:07 2018] LustreError: 161086:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880b00359200 x1604934219658192/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:431/0 lens 568/440 e 0 to 0 dl 1530794641 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 22:47:07 2018] LustreError: 161086:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 53 previous similar messages [Thu Jul 5 22:47:09 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Thu Jul 5 22:47:09 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 24 previous similar messages [Thu Jul 5 22:50:59 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 22:50:59 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 9584347 previous similar messages [Thu Jul 5 22:50:59 2018] Lustre: lustre-MDT0000: Client 4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad (at 172.16.229.39@o2ib) reconnecting [Thu Jul 5 22:50:59 2018] Lustre: Skipped 6879731 previous similar messages [Thu Jul 5 22:50:59 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Thu Jul 5 22:50:59 2018] Lustre: Skipped 6878602 previous similar messages [Thu Jul 5 22:51:10 2018] LustreError: 161086:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe317c50 x1604934246824224/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:639/0 lens 568/440 e 0 to 0 dl 1530794849 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 22:51:10 2018] LustreError: 161086:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 24 previous similar messages [Thu Jul 5 22:52:08 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8807a8ed4b00 x1604934253182128/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:734/0 lens 568/440 e 0 to 0 dl 1530794944 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 22:52:08 2018] LustreError: 4382:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 69 previous similar messages [Thu Jul 5 22:52:09 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Thu Jul 5 22:52:09 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 29 previous similar messages [Thu Jul 5 23:00:05 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8802df35d700 x1604934298827200/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:419/0 lens 568/440 e 0 to 0 dl 1530795384 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 23:00:05 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 25 previous similar messages [Thu Jul 5 23:00:59 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 23:00:59 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 9886264 previous similar messages [Thu Jul 5 23:00:59 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 23:00:59 2018] Lustre: Skipped 6792820 previous similar messages [Thu Jul 5 23:00:59 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 23:00:59 2018] Lustre: Skipped 6791753 previous similar messages [Thu Jul 5 23:02:09 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880409f86000 x1604934312543472/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:543/0 lens 568/440 e 0 to 0 dl 1530795508 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 23:02:09 2018] LustreError: 161088:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 65 previous similar messages [Thu Jul 5 23:02:16 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.229.39@o2ib [Thu Jul 5 23:02:16 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 28 previous similar messages [Thu Jul 5 23:10:59 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 23:10:59 2018] LustreError: 70022:0:(lod_dev.c:1414:lod_sync()) Skipped 9550225 previous similar messages [Thu Jul 5 23:10:59 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 23:10:59 2018] Lustre: Skipped 6853595 previous similar messages [Thu Jul 5 23:10:59 2018] Lustre: lustre-MDT0000: Connection restored to 0b6750c8-db73-a1e0-9386-cc5992fb3a68 (at 172.16.229.39@o2ib) [Thu Jul 5 23:10:59 2018] Lustre: Skipped 6852655 previous similar messages [Thu Jul 5 23:20:59 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 23:21:00 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 9859764 previous similar messages [Thu Jul 5 23:21:00 2018] Lustre: lustre-MDT0000: Client 4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad (at 172.16.229.39@o2ib) reconnecting [Thu Jul 5 23:21:00 2018] Lustre: Skipped 6661679 previous similar messages [Thu Jul 5 23:21:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 23:21:00 2018] Lustre: Skipped 6660731 previous similar messages [Thu Jul 5 23:24:07 2018] LustreError: 161086:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880606f49e00 x1604934447420368/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:388/0 lens 568/440 e 0 to 0 dl 1530796863 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Jul 5 23:24:07 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8803306b9e00 x1604934447420368/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:388/0 lens 568/440 e 0 to 0 dl 1530796863 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 23:24:07 2018] LustreError: 161088:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 29 previous similar messages [Thu Jul 5 23:24:07 2018] LustreError: 161086:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 17 previous similar messages [Thu Jul 5 23:24:55 2018] LustreError: 161085:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880f7c726f00 x1604934447420368/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:435/0 lens 568/440 e 0 to 0 dl 1530796910 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 23:24:55 2018] LustreError: 161085:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 6810 previous similar messages [Thu Jul 5 23:25:38 2018] LustreError: 161084:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8801b1399800 x1604934447420368/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:478/0 lens 568/440 e 0 to 0 dl 1530796953 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 23:25:38 2018] LustreError: 161084:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 163 previous similar messages [Thu Jul 5 23:25:40 2018] LustreError: 161083:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8806817d1b00 x1604934447420368/t0(0) o37->4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad@172.16.229.39@o2ib:481/0 lens 568/440 e 0 to 0 dl 1530796956 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Jul 5 23:25:40 2018] LustreError: 161083:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 36 previous similar messages [Thu Jul 5 23:26:59 2018] LNet: 162959:0:(o2iblnd_cb.c:2502:kiblnd_passive_connect()) Conn stale 172.16.229.39@o2ib version 12/12 incarnation 1530583738544115/1530796991349864 [Thu Jul 5 23:27:33 2018] LNet: Service thread pid 103699 was inactive for 200.56s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Jul 5 23:27:33 2018] Pid: 103699, comm: mdt01_021 [Thu Jul 5 23:27:33 2018] Call Trace: [Thu Jul 5 23:27:33 2018] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Thu Jul 5 23:27:33 2018] [] schedule+0x29/0x70 [Thu Jul 5 23:27:33 2018] [] schedule_timeout+0x174/0x2c0 [Thu Jul 5 23:27:33 2018] [] ? process_timeout+0x0/0x10 [Thu Jul 5 23:27:33 2018] [] ? ldlm_expired_completion_wait+0x0/0x220 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] ? default_wake_function+0x0/0x20 [Thu Jul 5 23:27:33 2018] [] ldlm_cli_enqueue_local+0x230/0x850 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] [Thu Jul 5 23:27:33 2018] [] mdt_object_local_lock+0x3fc/0xae0 [mdt] [Thu Jul 5 23:27:33 2018] [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] [Thu Jul 5 23:27:33 2018] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] mdt_object_lock_internal+0x70/0x330 [mdt] [Thu Jul 5 23:27:33 2018] [] mdt_object_lock+0x20/0x30 [mdt] [Thu Jul 5 23:27:33 2018] [] mdt_reint_open+0xda9/0x3260 [mdt] [Thu Jul 5 23:27:33 2018] [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] [Thu Jul 5 23:27:33 2018] [] ? ucred_set_jobid+0x53/0x70 [mdt] [Thu Jul 5 23:27:33 2018] [] mdt_reint_rec+0x80/0x210 [mdt] [Thu Jul 5 23:27:33 2018] [] mdt_reint_internal+0x5fb/0x9c0 [mdt] [Thu Jul 5 23:27:33 2018] [] mdt_intent_reint+0x157/0x420 [mdt] [Thu Jul 5 23:27:33 2018] [] mdt_intent_opc+0x442/0xad0 [mdt] [Thu Jul 5 23:27:33 2018] [] ? lustre_swab_ldlm_intent+0x0/0x20 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] mdt_intent_policy+0x1a3/0x360 [mdt] [Thu Jul 5 23:27:33 2018] [] ldlm_lock_enqueue+0x382/0x8f0 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] ldlm_handle_enqueue0+0x8f3/0x13e0 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] tgt_request_handle+0x925/0x13b0 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] ? default_wake_function+0x12/0x20 [Thu Jul 5 23:27:33 2018] [] ? __wake_up_common+0x58/0x90 [Thu Jul 5 23:27:33 2018] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [Thu Jul 5 23:27:33 2018] [] kthread+0xcf/0xe0 [Thu Jul 5 23:27:33 2018] [] ? kthread+0x0/0xe0 [Thu Jul 5 23:27:33 2018] [] ret_from_fork+0x58/0x90 [Thu Jul 5 23:27:33 2018] [] ? kthread+0x0/0xe0 [Thu Jul 5 23:27:33 2018] LustreError: dumping log to /tmp/lustre-log.1530797026.103699 [Thu Jul 5 23:28:16 2018] LNet: Service thread pid 4631 was inactive for 212.73s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Jul 5 23:28:16 2018] Pid: 4631, comm: mdt00_003 [Thu Jul 5 23:28:16 2018] Call Trace: [Thu Jul 5 23:28:16 2018] [] ? load_balance+0x192/0x9a0 [Thu Jul 5 23:28:16 2018] [] schedule+0x29/0x70 [Thu Jul 5 23:28:16 2018] [] schedule_timeout+0x174/0x2c0 [Thu Jul 5 23:28:16 2018] [] ? process_timeout+0x0/0x10 [Thu Jul 5 23:28:16 2018] [] ? ldlm_expired_completion_wait+0x0/0x220 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] ? default_wake_function+0x0/0x20 [Thu Jul 5 23:28:16 2018] [] ldlm_cli_enqueue_local+0x230/0x850 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] [Thu Jul 5 23:28:16 2018] [] mdt_object_local_lock+0x3fc/0xae0 [mdt] [Thu Jul 5 23:28:16 2018] [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] [Thu Jul 5 23:28:16 2018] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] mdt_object_lock_internal+0x70/0x330 [mdt] [Thu Jul 5 23:28:16 2018] [] mdt_object_lock+0x20/0x30 [mdt] [Thu Jul 5 23:28:16 2018] [] mdt_reint_open+0xda9/0x3260 [mdt] [Thu Jul 5 23:28:16 2018] [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] [Thu Jul 5 23:28:16 2018] [] ? ucred_set_jobid+0x53/0x70 [mdt] [Thu Jul 5 23:28:16 2018] [] mdt_reint_rec+0x80/0x210 [mdt] [Thu Jul 5 23:28:16 2018] [] mdt_reint_internal+0x5fb/0x9c0 [mdt] [Thu Jul 5 23:28:16 2018] [] mdt_intent_reint+0x157/0x420 [mdt] [Thu Jul 5 23:28:16 2018] [] mdt_intent_opc+0x442/0xad0 [mdt] [Thu Jul 5 23:28:16 2018] [] ? lustre_swab_ldlm_intent+0x0/0x20 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] mdt_intent_policy+0x1a3/0x360 [mdt] [Thu Jul 5 23:28:16 2018] [] ldlm_lock_enqueue+0x382/0x8f0 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] ldlm_handle_enqueue0+0x8f3/0x13e0 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] tgt_request_handle+0x925/0x13b0 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] ? default_wake_function+0x12/0x20 [Thu Jul 5 23:28:16 2018] [] ? __wake_up_common+0x58/0x90 [Thu Jul 5 23:28:16 2018] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [Thu Jul 5 23:28:16 2018] [] kthread+0xcf/0xe0 [Thu Jul 5 23:28:16 2018] [] ? kthread+0x0/0xe0 [Thu Jul 5 23:28:16 2018] [] ret_from_fork+0x58/0x90 [Thu Jul 5 23:28:16 2018] [] ? kthread+0x0/0xe0 [Thu Jul 5 23:28:16 2018] LustreError: dumping log to /tmp/lustre-log.1530797069.4631 [Thu Jul 5 23:28:17 2018] Pid: 5212, comm: mdt00_005 [Thu Jul 5 23:28:17 2018] Call Trace: [Thu Jul 5 23:28:17 2018] [] schedule+0x29/0x70 [Thu Jul 5 23:28:17 2018] [] schedule_timeout+0x174/0x2c0 [Thu Jul 5 23:28:17 2018] [] ? process_timeout+0x0/0x10 [Thu Jul 5 23:28:17 2018] [] ? ldlm_expired_completion_wait+0x0/0x220 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] ? default_wake_function+0x0/0x20 [Thu Jul 5 23:28:17 2018] [] ldlm_cli_enqueue_local+0x230/0x850 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] [Thu Jul 5 23:28:17 2018] [] mdt_object_local_lock+0x3fc/0xae0 [mdt] [Thu Jul 5 23:28:17 2018] [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] [Thu Jul 5 23:28:17 2018] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] mdt_object_lock_internal+0x70/0x330 [mdt] [Thu Jul 5 23:28:17 2018] [] mdt_object_lock+0x20/0x30 [mdt] [Thu Jul 5 23:28:17 2018] [] mdt_reint_open+0xda9/0x3260 [mdt] [Thu Jul 5 23:28:17 2018] [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] [Thu Jul 5 23:28:17 2018] [] ? ucred_set_jobid+0x53/0x70 [mdt] [Thu Jul 5 23:28:17 2018] [] mdt_reint_rec+0x80/0x210 [mdt] [Thu Jul 5 23:28:17 2018] [] mdt_reint_internal+0x5fb/0x9c0 [mdt] [Thu Jul 5 23:28:17 2018] [] mdt_intent_reint+0x157/0x420 [mdt] [Thu Jul 5 23:28:17 2018] [] mdt_intent_opc+0x442/0xad0 [mdt] [Thu Jul 5 23:28:17 2018] [] ? lustre_swab_ldlm_intent+0x0/0x20 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] mdt_intent_policy+0x1a3/0x360 [mdt] [Thu Jul 5 23:28:17 2018] [] ldlm_lock_enqueue+0x382/0x8f0 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] ldlm_handle_enqueue0+0x8f3/0x13e0 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] tgt_request_handle+0x925/0x13b0 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] ? default_wake_function+0x12/0x20 [Thu Jul 5 23:28:17 2018] [] ? __wake_up_common+0x58/0x90 [Thu Jul 5 23:28:17 2018] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [Thu Jul 5 23:28:17 2018] [] kthread+0xcf/0xe0 [Thu Jul 5 23:28:17 2018] [] ? kthread+0x0/0xe0 [Thu Jul 5 23:28:17 2018] [] ret_from_fork+0x58/0x90 [Thu Jul 5 23:28:17 2018] [] ? kthread+0x0/0xe0 [Thu Jul 5 23:29:13 2018] LustreError: 103699:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1530796825, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-lustre-MDT0000_UUID lock: ffff881570a51f80/0x88fac5ef447faef lrc: 3/0,1 mode: --/CW res: [0x200009cb9:0xaf48:0x0].0x0 bits 0x2/0x0 rrc: 15 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 103699 timeout: 0 lvb_type: 0 [Thu Jul 5 23:29:13 2018] LustreError: dumping log to /tmp/lustre-log.1530797125.103699 [Thu Jul 5 23:29:15 2018] Lustre: MGS: haven't heard from client 5e77915c-73b3-4cab-6399-5b6adf3c471e (at 172.16.229.39@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102e20b000, cur 1530797128 expire 1530796978 last 1530796901 [Thu Jul 5 23:29:38 2018] Lustre: lustre-MDT0000: haven't heard from client 4922dce1-5cb6-82f2-5f0f-bb5ace2ce4ad (at 172.16.229.39@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ff61a7000, cur 1530797150 expire 1530797000 last 1530796923 [Thu Jul 5 23:29:38 2018] LNet: Service thread pid 5212 completed after 293.80s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Thu Jul 5 23:29:38 2018] LNet: Skipped 1 previous similar message [Thu Jul 5 23:31:00 2018] LustreError: 4631:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 23:31:00 2018] LustreError: 4631:0:(lod_dev.c:1414:lod_sync()) Skipped 6907487 previous similar messages [Thu Jul 5 23:31:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 23:31:00 2018] Lustre: Skipped 4431303 previous similar messages [Thu Jul 5 23:31:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 23:31:00 2018] Lustre: Skipped 4430783 previous similar messages [Thu Jul 5 23:36:47 2018] in:imjournal[35772]: segfault at 0 ip 00007f88d84f7983 sp 00007f88d1414b50 error 4 in imjournal.so[7f88d84f5000+5000] [Thu Jul 5 23:41:00 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 23:41:00 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) Skipped 4933749 previous similar messages [Thu Jul 5 23:41:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 23:41:00 2018] Lustre: Skipped 2466931 previous similar messages [Thu Jul 5 23:41:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 23:41:00 2018] Lustre: Skipped 2466931 previous similar messages [Thu Jul 5 23:51:00 2018] LustreError: 60612:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Thu Jul 5 23:51:00 2018] LustreError: 60612:0:(lod_dev.c:1414:lod_sync()) Skipped 4003840 previous similar messages [Thu Jul 5 23:51:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Thu Jul 5 23:51:00 2018] Lustre: Skipped 2001973 previous similar messages [Thu Jul 5 23:51:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Thu Jul 5 23:51:00 2018] Lustre: Skipped 2001973 previous similar messages [Fri Jul 6 00:01:00 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 00:01:00 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 5432746 previous similar messages [Fri Jul 6 00:01:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 00:01:00 2018] Lustre: Skipped 2716634 previous similar messages [Fri Jul 6 00:01:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 00:01:00 2018] Lustre: Skipped 2716634 previous similar messages [Fri Jul 6 00:11:00 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 00:11:00 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) Skipped 5274793 previous similar messages [Fri Jul 6 00:11:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 00:11:00 2018] Lustre: Skipped 2637545 previous similar messages [Fri Jul 6 00:11:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 00:11:00 2018] Lustre: Skipped 2637545 previous similar messages [Fri Jul 6 00:21:00 2018] LustreError: 4375:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 00:21:00 2018] LustreError: 4375:0:(lod_dev.c:1414:lod_sync()) Skipped 5238782 previous similar messages [Fri Jul 6 00:21:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 00:21:00 2018] Lustre: Skipped 2619537 previous similar messages [Fri Jul 6 00:21:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 00:21:00 2018] Lustre: Skipped 2619537 previous similar messages [Fri Jul 6 00:31:00 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 00:31:00 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 5217361 previous similar messages [Fri Jul 6 00:31:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 00:31:00 2018] Lustre: Skipped 2608838 previous similar messages [Fri Jul 6 00:31:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 00:31:00 2018] Lustre: Skipped 2608838 previous similar messages [Fri Jul 6 00:41:00 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 00:41:00 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 5153896 previous similar messages [Fri Jul 6 00:41:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 00:41:00 2018] Lustre: Skipped 2577037 previous similar messages [Fri Jul 6 00:41:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 00:41:00 2018] Lustre: Skipped 2577037 previous similar messages [Fri Jul 6 00:51:00 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 00:51:00 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) Skipped 4757486 previous similar messages [Fri Jul 6 00:51:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 00:51:00 2018] Lustre: Skipped 2612535 previous similar messages [Fri Jul 6 00:51:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 00:51:00 2018] Lustre: Skipped 2612535 previous similar messages [Fri Jul 6 01:01:00 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 01:01:00 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 5255979 previous similar messages [Fri Jul 6 01:01:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 01:01:00 2018] Lustre: Skipped 2628042 previous similar messages [Fri Jul 6 01:01:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 01:01:00 2018] Lustre: Skipped 2628042 previous similar messages [Fri Jul 6 01:11:00 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 01:11:00 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 5495725 previous similar messages [Fri Jul 6 01:11:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 01:11:00 2018] Lustre: Skipped 2747985 previous similar messages [Fri Jul 6 01:11:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 01:11:00 2018] Lustre: Skipped 2747985 previous similar messages [Fri Jul 6 01:21:00 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 01:21:00 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) Skipped 6630084 previous similar messages [Fri Jul 6 01:21:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 01:21:00 2018] Lustre: Skipped 3315344 previous similar messages [Fri Jul 6 01:21:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 01:21:00 2018] Lustre: Skipped 3315344 previous similar messages [Fri Jul 6 01:31:00 2018] LustreError: 5022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 01:31:00 2018] LustreError: 5022:0:(lod_dev.c:1414:lod_sync()) Skipped 6377898 previous similar messages [Fri Jul 6 01:31:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 01:31:00 2018] Lustre: Skipped 3396729 previous similar messages [Fri Jul 6 01:31:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 01:31:00 2018] Lustre: Skipped 3396729 previous similar messages [Fri Jul 6 01:41:00 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 01:41:00 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 4952784 previous similar messages [Fri Jul 6 01:41:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 01:41:00 2018] Lustre: Skipped 2555569 previous similar messages [Fri Jul 6 01:41:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 01:41:00 2018] Lustre: Skipped 2555569 previous similar messages [Fri Jul 6 01:51:00 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 01:51:00 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 4204534 previous similar messages [Fri Jul 6 01:51:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 01:51:00 2018] Lustre: Skipped 2102326 previous similar messages [Fri Jul 6 01:51:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 01:51:00 2018] Lustre: Skipped 2102326 previous similar messages [Fri Jul 6 02:01:00 2018] LustreError: 4375:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 02:01:00 2018] LustreError: 4375:0:(lod_dev.c:1414:lod_sync()) Skipped 6684987 previous similar messages [Fri Jul 6 02:01:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 02:01:00 2018] Lustre: Skipped 3342715 previous similar messages [Fri Jul 6 02:01:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 02:01:00 2018] Lustre: Skipped 3342715 previous similar messages [Fri Jul 6 02:11:00 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 02:11:00 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) Skipped 6654794 previous similar messages [Fri Jul 6 02:11:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 02:11:00 2018] Lustre: Skipped 3327654 previous similar messages [Fri Jul 6 02:11:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 02:11:00 2018] Lustre: Skipped 3327654 previous similar messages [Fri Jul 6 02:21:00 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 02:21:00 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) Skipped 6548922 previous similar messages [Fri Jul 6 02:21:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 02:21:00 2018] Lustre: Skipped 3274673 previous similar messages [Fri Jul 6 02:21:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 02:21:00 2018] Lustre: Skipped 3274673 previous similar messages [Fri Jul 6 02:31:00 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 02:31:00 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 6430688 previous similar messages [Fri Jul 6 02:31:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 02:31:00 2018] Lustre: Skipped 3215692 previous similar messages [Fri Jul 6 02:31:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 02:31:00 2018] Lustre: Skipped 3215692 previous similar messages [Fri Jul 6 02:41:00 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 02:41:00 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) Skipped 5542004 previous similar messages [Fri Jul 6 02:41:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 02:41:00 2018] Lustre: Skipped 2771084 previous similar messages [Fri Jul 6 02:41:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 02:41:00 2018] Lustre: Skipped 2771084 previous similar messages [Fri Jul 6 02:51:00 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 02:51:00 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 4964814 previous similar messages [Fri Jul 6 02:51:00 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 02:51:00 2018] Lustre: Skipped 2482736 previous similar messages [Fri Jul 6 02:51:00 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 02:51:00 2018] Lustre: Skipped 2482736 previous similar messages [Fri Jul 6 03:01:00 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 03:01:00 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 6159423 previous similar messages [Fri Jul 6 03:01:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 03:01:01 2018] Lustre: Skipped 3357354 previous similar messages [Fri Jul 6 03:01:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 03:01:01 2018] Lustre: Skipped 3357354 previous similar messages [Fri Jul 6 03:11:01 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 03:11:01 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) Skipped 6036063 previous similar messages [Fri Jul 6 03:11:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 03:11:01 2018] Lustre: Skipped 3330758 previous similar messages [Fri Jul 6 03:11:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 03:11:01 2018] Lustre: Skipped 3330758 previous similar messages [Fri Jul 6 03:21:01 2018] LustreError: 4631:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 03:21:01 2018] LustreError: 4631:0:(lod_dev.c:1414:lod_sync()) Skipped 5488735 previous similar messages [Fri Jul 6 03:21:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 03:21:01 2018] Lustre: Skipped 2744615 previous similar messages [Fri Jul 6 03:21:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 03:21:01 2018] Lustre: Skipped 2744615 previous similar messages [Fri Jul 6 03:31:01 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 03:31:01 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) Skipped 5950834 previous similar messages [Fri Jul 6 03:31:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 03:31:01 2018] Lustre: Skipped 2975446 previous similar messages [Fri Jul 6 03:31:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 03:31:01 2018] Lustre: Skipped 2975446 previous similar messages [Fri Jul 6 03:41:01 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 03:41:01 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 6390271 previous similar messages [Fri Jul 6 03:41:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 03:41:01 2018] Lustre: Skipped 3195344 previous similar messages [Fri Jul 6 03:41:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 03:41:01 2018] Lustre: Skipped 3195344 previous similar messages [Fri Jul 6 03:51:01 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 03:51:01 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 5355798 previous similar messages [Fri Jul 6 03:51:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 03:51:01 2018] Lustre: Skipped 2678199 previous similar messages [Fri Jul 6 03:51:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 03:51:01 2018] Lustre: Skipped 2678199 previous similar messages [Fri Jul 6 04:01:01 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 04:01:01 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 6757347 previous similar messages [Fri Jul 6 04:01:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 04:01:01 2018] Lustre: Skipped 3378818 previous similar messages [Fri Jul 6 04:01:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 04:01:01 2018] Lustre: Skipped 3378818 previous similar messages [Fri Jul 6 04:11:01 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 04:11:01 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 5211869 previous similar messages [Fri Jul 6 04:11:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 04:11:01 2018] Lustre: Skipped 2605949 previous similar messages [Fri Jul 6 04:11:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 04:11:01 2018] Lustre: Skipped 2605949 previous similar messages [Fri Jul 6 04:21:01 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 04:21:01 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 5441451 previous similar messages [Fri Jul 6 04:21:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 04:21:01 2018] Lustre: Skipped 2720777 previous similar messages [Fri Jul 6 04:21:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 04:21:01 2018] Lustre: Skipped 2720777 previous similar messages [Fri Jul 6 04:31:01 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 04:31:01 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 6355824 previous similar messages [Fri Jul 6 04:31:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 04:31:01 2018] Lustre: Skipped 3177992 previous similar messages [Fri Jul 6 04:31:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 04:31:01 2018] Lustre: Skipped 3177992 previous similar messages [Fri Jul 6 04:41:01 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 04:41:01 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 6488707 previous similar messages [Fri Jul 6 04:41:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 04:41:01 2018] Lustre: Skipped 3244447 previous similar messages [Fri Jul 6 04:41:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 04:41:01 2018] Lustre: Skipped 3244447 previous similar messages [Fri Jul 6 04:51:01 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 04:51:01 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 4861972 previous similar messages [Fri Jul 6 04:51:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 04:51:01 2018] Lustre: Skipped 2431308 previous similar messages [Fri Jul 6 04:51:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 04:51:01 2018] Lustre: Skipped 2431308 previous similar messages [Fri Jul 6 05:01:01 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 05:01:01 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 4806197 previous similar messages [Fri Jul 6 05:01:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 05:01:01 2018] Lustre: Skipped 2403140 previous similar messages [Fri Jul 6 05:01:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 05:01:01 2018] Lustre: Skipped 2403140 previous similar messages [Fri Jul 6 05:11:01 2018] LustreError: 5022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 05:11:01 2018] LustreError: 5022:0:(lod_dev.c:1414:lod_sync()) Skipped 5126308 previous similar messages [Fri Jul 6 05:11:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 05:11:01 2018] Lustre: Skipped 2563675 previous similar messages [Fri Jul 6 05:11:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 05:11:01 2018] Lustre: Skipped 2563675 previous similar messages [Fri Jul 6 05:21:01 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 05:21:01 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) Skipped 5297073 previous similar messages [Fri Jul 6 05:21:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 05:21:01 2018] Lustre: Skipped 2648729 previous similar messages [Fri Jul 6 05:21:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 05:21:01 2018] Lustre: Skipped 2648729 previous similar messages [Fri Jul 6 05:31:01 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 05:31:01 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) Skipped 4734715 previous similar messages [Fri Jul 6 05:31:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 05:31:01 2018] Lustre: Skipped 2367463 previous similar messages [Fri Jul 6 05:31:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 05:31:01 2018] Lustre: Skipped 2367463 previous similar messages [Fri Jul 6 05:41:01 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 05:41:01 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 5303749 previous similar messages [Fri Jul 6 05:41:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 05:41:01 2018] Lustre: Skipped 2652343 previous similar messages [Fri Jul 6 05:41:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 05:41:01 2018] Lustre: Skipped 2652343 previous similar messages [Fri Jul 6 05:51:01 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 05:51:01 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) Skipped 4878182 previous similar messages [Fri Jul 6 05:51:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 05:51:01 2018] Lustre: Skipped 2439330 previous similar messages [Fri Jul 6 05:51:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 05:51:01 2018] Lustre: Skipped 2439330 previous similar messages [Fri Jul 6 06:01:01 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 06:01:01 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 4705964 previous similar messages [Fri Jul 6 06:01:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 06:01:01 2018] Lustre: Skipped 2353218 previous similar messages [Fri Jul 6 06:01:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 06:01:01 2018] Lustre: Skipped 2353218 previous similar messages [Fri Jul 6 06:11:01 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 06:11:01 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 4374539 previous similar messages [Fri Jul 6 06:11:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 06:11:01 2018] Lustre: Skipped 2187636 previous similar messages [Fri Jul 6 06:11:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 06:11:01 2018] Lustre: Skipped 2187636 previous similar messages [Fri Jul 6 06:21:01 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 06:21:01 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) Skipped 6427689 previous similar messages [Fri Jul 6 06:21:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 06:21:01 2018] Lustre: Skipped 3214172 previous similar messages [Fri Jul 6 06:21:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 06:21:01 2018] Lustre: Skipped 3214172 previous similar messages [Fri Jul 6 06:31:01 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 06:31:01 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 6024791 previous similar messages [Fri Jul 6 06:31:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 06:31:01 2018] Lustre: Skipped 3012551 previous similar messages [Fri Jul 6 06:31:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 06:31:01 2018] Lustre: Skipped 3012551 previous similar messages [Fri Jul 6 06:41:01 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 06:41:01 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) Skipped 4628675 previous similar messages [Fri Jul 6 06:41:01 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 06:41:01 2018] Lustre: Skipped 2314438 previous similar messages [Fri Jul 6 06:41:01 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 06:41:01 2018] Lustre: Skipped 2314438 previous similar messages [Fri Jul 6 06:51:02 2018] LustreError: 4375:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 06:51:02 2018] LustreError: 4375:0:(lod_dev.c:1414:lod_sync()) Skipped 6026904 previous similar messages [Fri Jul 6 06:51:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 06:51:02 2018] Lustre: Skipped 3013537 previous similar messages [Fri Jul 6 06:51:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 06:51:02 2018] Lustre: Skipped 3013537 previous similar messages [Fri Jul 6 07:01:02 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 07:01:02 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 5230665 previous similar messages [Fri Jul 6 07:01:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 07:01:02 2018] Lustre: Skipped 2615601 previous similar messages [Fri Jul 6 07:01:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 07:01:02 2018] Lustre: Skipped 2615601 previous similar messages [Fri Jul 6 07:11:02 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 07:11:02 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 4630995 previous similar messages [Fri Jul 6 07:11:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 07:11:02 2018] Lustre: Skipped 2315535 previous similar messages [Fri Jul 6 07:11:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 07:11:02 2018] Lustre: Skipped 2315535 previous similar messages [Fri Jul 6 07:21:02 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 07:21:02 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 4882036 previous similar messages [Fri Jul 6 07:21:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 07:21:02 2018] Lustre: Skipped 2441491 previous similar messages [Fri Jul 6 07:21:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 07:21:02 2018] Lustre: Skipped 2441491 previous similar messages [Fri Jul 6 07:31:02 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 07:31:02 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) Skipped 4726217 previous similar messages [Fri Jul 6 07:31:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 07:31:02 2018] Lustre: Skipped 2363316 previous similar messages [Fri Jul 6 07:31:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 07:31:02 2018] Lustre: Skipped 2363316 previous similar messages [Fri Jul 6 07:41:02 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 07:41:02 2018] LustreError: 89834:0:(lod_dev.c:1414:lod_sync()) Skipped 6372907 previous similar messages [Fri Jul 6 07:41:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 07:41:02 2018] Lustre: Skipped 3186768 previous similar messages [Fri Jul 6 07:41:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 07:41:02 2018] Lustre: Skipped 3186768 previous similar messages [Fri Jul 6 07:51:02 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 07:51:02 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 6424146 previous similar messages [Fri Jul 6 07:51:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 07:51:02 2018] Lustre: Skipped 3212264 previous similar messages [Fri Jul 6 07:51:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 07:51:02 2018] Lustre: Skipped 3212264 previous similar messages [Fri Jul 6 08:01:02 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 08:01:02 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) Skipped 6400173 previous similar messages [Fri Jul 6 08:01:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 08:01:02 2018] Lustre: Skipped 3200533 previous similar messages [Fri Jul 6 08:01:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 08:01:02 2018] Lustre: Skipped 3200533 previous similar messages [Fri Jul 6 08:11:02 2018] LustreError: 60612:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 08:11:02 2018] LustreError: 60612:0:(lod_dev.c:1414:lod_sync()) Skipped 4598748 previous similar messages [Fri Jul 6 08:11:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 08:11:02 2018] Lustre: Skipped 2299607 previous similar messages [Fri Jul 6 08:11:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 08:11:02 2018] Lustre: Skipped 2299607 previous similar messages [Fri Jul 6 08:21:02 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 08:21:02 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 4744885 previous similar messages [Fri Jul 6 08:21:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 08:21:02 2018] Lustre: Skipped 2372603 previous similar messages [Fri Jul 6 08:21:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 08:21:02 2018] Lustre: Skipped 2372603 previous similar messages [Fri Jul 6 08:31:02 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 08:31:02 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) Skipped 5011981 previous similar messages [Fri Jul 6 08:31:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 08:31:02 2018] Lustre: Skipped 2613319 previous similar messages [Fri Jul 6 08:31:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 08:31:02 2018] Lustre: Skipped 2613319 previous similar messages [Fri Jul 6 08:41:02 2018] LustreError: 5022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 08:41:02 2018] LustreError: 5022:0:(lod_dev.c:1414:lod_sync()) Skipped 4668072 previous similar messages [Fri Jul 6 08:41:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 08:41:02 2018] Lustre: Skipped 2334501 previous similar messages [Fri Jul 6 08:41:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 08:41:02 2018] Lustre: Skipped 2334501 previous similar messages [Fri Jul 6 08:51:02 2018] LustreError: 4375:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 08:51:02 2018] LustreError: 4375:0:(lod_dev.c:1414:lod_sync()) Skipped 5444470 previous similar messages [Fri Jul 6 08:51:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 08:51:02 2018] Lustre: Skipped 2722662 previous similar messages [Fri Jul 6 08:51:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 08:51:02 2018] Lustre: Skipped 2722662 previous similar messages [Fri Jul 6 09:01:02 2018] LustreError: 5022:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 09:01:02 2018] LustreError: 5022:0:(lod_dev.c:1414:lod_sync()) Skipped 5020995 previous similar messages [Fri Jul 6 09:01:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 09:01:02 2018] Lustre: Skipped 2510696 previous similar messages [Fri Jul 6 09:01:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 09:01:02 2018] Lustre: Skipped 2510696 previous similar messages [Fri Jul 6 09:11:02 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 09:11:02 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 7797752 previous similar messages [Fri Jul 6 09:11:02 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 09:11:02 2018] Lustre: Skipped 4756143 previous similar messages [Fri Jul 6 09:11:02 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 09:11:02 2018] Lustre: Skipped 4755694 previous similar messages [Fri Jul 6 09:21:02 2018] LustreError: 4375:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 09:21:02 2018] LustreError: 4375:0:(lod_dev.c:1414:lod_sync()) Skipped 8114622 previous similar messages [Fri Jul 6 09:21:02 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 09:21:02 2018] Lustre: Skipped 5300143 previous similar messages [Fri Jul 6 09:21:02 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 09:21:02 2018] Lustre: Skipped 5299542 previous similar messages [Fri Jul 6 09:31:02 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 09:31:02 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 8170801 previous similar messages [Fri Jul 6 09:31:02 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 09:31:02 2018] Lustre: Skipped 5348596 previous similar messages [Fri Jul 6 09:31:02 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 09:31:02 2018] Lustre: Skipped 5347802 previous similar messages [Fri Jul 6 09:41:02 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 09:41:02 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 7863823 previous similar messages [Fri Jul 6 09:41:02 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 09:41:02 2018] Lustre: Skipped 5028553 previous similar messages [Fri Jul 6 09:41:02 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 09:41:02 2018] Lustre: Skipped 5027938 previous similar messages [Fri Jul 6 09:51:02 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 09:51:02 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 8449422 previous similar messages [Fri Jul 6 09:51:02 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 09:51:02 2018] Lustre: Skipped 5544700 previous similar messages [Fri Jul 6 09:51:02 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 09:51:02 2018] Lustre: Skipped 5543727 previous similar messages [Fri Jul 6 10:01:02 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 10:01:02 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 9072786 previous similar messages [Fri Jul 6 10:01:02 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 10:01:02 2018] Lustre: Skipped 5792369 previous similar messages [Fri Jul 6 10:01:02 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 10:01:02 2018] Lustre: Skipped 5791570 previous similar messages [Fri Jul 6 10:04:21 2018] LustreError: 73113:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880bc8a07b00 x1603047683006528/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:257/0 lens 568/440 e 0 to 0 dl 1530835237 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 10:04:21 2018] LustreError: 161083:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8808cc669500 x1603047683006528/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:257/0 lens 568/440 e 0 to 0 dl 1530835237 ref 1 fl Interpret:/2/0 rc 0/0 [Fri Jul 6 10:04:21 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.230.53@o2ib [Fri Jul 6 10:04:21 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 15 previous similar messages [Fri Jul 6 10:04:21 2018] LustreError: 73113:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 8781 previous similar messages [Fri Jul 6 10:04:31 2018] LustreError: 161085:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ff6a32d00 x1603047683663264/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:267/0 lens 568/440 e 0 to 0 dl 1530835247 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 10:04:31 2018] LustreError: 161085:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 8 previous similar messages [Fri Jul 6 10:04:37 2018] LustreError: 161085:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880713611e00 x1603047684037136/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:272/0 lens 568/440 e 0 to 0 dl 1530835252 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 10:04:37 2018] LustreError: 161085:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Fri Jul 6 10:04:52 2018] LustreError: 73113:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffe314450 x1603047685042288/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:288/0 lens 568/440 e 0 to 0 dl 1530835268 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 10:04:52 2018] LustreError: 161085:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88049732fb00 x1603047685042288/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:288/0 lens 568/440 e 0 to 0 dl 1530835268 ref 1 fl Interpret:/2/0 rc 0/0 [Fri Jul 6 10:04:52 2018] LustreError: 161085:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 14 previous similar messages [Fri Jul 6 10:04:52 2018] LustreError: 73113:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Fri Jul 6 10:11:02 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 10:11:02 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 7352750 previous similar messages [Fri Jul 6 10:11:02 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 10:11:02 2018] Lustre: Skipped 4839922 previous similar messages [Fri Jul 6 10:11:02 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 10:11:02 2018] Lustre: Skipped 4839491 previous similar messages [Fri Jul 6 10:21:02 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 10:21:02 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 7573802 previous similar messages [Fri Jul 6 10:21:02 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 10:21:02 2018] Lustre: Skipped 5016514 previous similar messages [Fri Jul 6 10:21:02 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 10:21:02 2018] Lustre: Skipped 5015690 previous similar messages [Fri Jul 6 10:25:38 2018] LustreError: 73113:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880ffd2f9050 x1603047766721600/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:24/0 lens 568/440 e 0 to 0 dl 1530836514 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 10:25:38 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8804c3a59200 x1603047766721600/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:24/0 lens 568/440 e 0 to 0 dl 1530836514 ref 1 fl Interpret:/2/0 rc 0/0 [Fri Jul 6 10:25:38 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 13 previous similar messages [Fri Jul 6 10:25:38 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.230.53@o2ib [Fri Jul 6 10:25:38 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 15 previous similar messages [Fri Jul 6 10:25:38 2018] LustreError: 73113:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 2 previous similar messages [Fri Jul 6 10:25:44 2018] LustreError: 161085:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8808be652700 x1603047767115344/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:30/0 lens 568/440 e 0 to 0 dl 1530836520 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 10:25:44 2018] LustreError: 73113:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8808d13ad100 x1603047767115344/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:30/0 lens 568/440 e 0 to 0 dl 1530836520 ref 1 fl Interpret:/2/0 rc 0/0 [Fri Jul 6 10:25:44 2018] LustreError: 73113:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 5 previous similar messages [Fri Jul 6 10:25:50 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.230.53@o2ib [Fri Jul 6 10:25:50 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 3 previous similar messages [Fri Jul 6 10:25:54 2018] LustreError: 161085:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8808b6b67200 x1603047767693744/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:64/0 lens 568/440 e 1 to 0 dl 1530836554 ref 1 fl Interpret:/2/0 rc 0/0 [Fri Jul 6 10:25:54 2018] LustreError: 161085:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 7 previous similar messages [Fri Jul 6 10:26:11 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.230.53@o2ib [Fri Jul 6 10:26:11 2018] LNet: 2853:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 5 previous similar messages [Fri Jul 6 10:26:13 2018] LustreError: 73113:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88087a746c00 x1603047768934688/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:95/0 lens 568/440 e 0 to 0 dl 1530836585 ref 1 fl Interpret:/2/0 rc 0/0 [Fri Jul 6 10:26:13 2018] LustreError: 73113:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 10 previous similar messages [Fri Jul 6 10:26:14 2018] LustreError: 73113:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880119ea9e00 x1603047769064496/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:97/0 lens 568/440 e 0 to 0 dl 1530836587 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 10:26:26 2018] LustreError: 161083:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880b13ebd400 x1603047769860768/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:109/0 lens 568/440 e 0 to 0 dl 1530836599 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 10:26:26 2018] LustreError: 161083:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 1 previous similar message [Fri Jul 6 10:31:02 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 10:31:02 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 5586169 previous similar messages [Fri Jul 6 10:31:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 10:31:03 2018] Lustre: Skipped 3680987 previous similar messages [Fri Jul 6 10:31:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 10:31:03 2018] Lustre: Skipped 3680074 previous similar messages [Fri Jul 6 10:41:03 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 10:41:03 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 7111407 previous similar messages [Fri Jul 6 10:41:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 10:41:03 2018] Lustre: Skipped 4800502 previous similar messages [Fri Jul 6 10:41:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 10:41:03 2018] Lustre: Skipped 4799416 previous similar messages [Fri Jul 6 10:51:03 2018] LustreError: 60612:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 10:51:03 2018] LustreError: 60612:0:(lod_dev.c:1414:lod_sync()) Skipped 7386379 previous similar messages [Fri Jul 6 10:51:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 10:51:03 2018] Lustre: Skipped 5001313 previous similar messages [Fri Jul 6 10:51:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 10:51:03 2018] Lustre: Skipped 5000395 previous similar messages [Fri Jul 6 11:01:03 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 11:01:03 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 8511211 previous similar messages [Fri Jul 6 11:01:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 11:01:03 2018] Lustre: Skipped 5402022 previous similar messages [Fri Jul 6 11:01:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 11:01:03 2018] Lustre: Skipped 5401558 previous similar messages [Fri Jul 6 11:11:03 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 11:11:03 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 7750682 previous similar messages [Fri Jul 6 11:11:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 11:11:03 2018] Lustre: Skipped 5238692 previous similar messages [Fri Jul 6 11:11:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 11:11:03 2018] Lustre: Skipped 5237158 previous similar messages [Fri Jul 6 11:21:03 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 11:21:03 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 7838873 previous similar messages [Fri Jul 6 11:21:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 11:21:03 2018] Lustre: Skipped 5052893 previous similar messages [Fri Jul 6 11:21:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 11:21:03 2018] Lustre: Skipped 5051837 previous similar messages [Fri Jul 6 11:31:03 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 11:31:03 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 8955990 previous similar messages [Fri Jul 6 11:31:03 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 11:31:03 2018] Lustre: Skipped 5782013 previous similar messages [Fri Jul 6 11:31:03 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 11:31:03 2018] Lustre: Skipped 5781529 previous similar messages [Fri Jul 6 11:41:03 2018] LustreError: 4375:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 11:41:03 2018] LustreError: 4375:0:(lod_dev.c:1414:lod_sync()) Skipped 7586851 previous similar messages [Fri Jul 6 11:41:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 11:41:03 2018] Lustre: Skipped 4806349 previous similar messages [Fri Jul 6 11:41:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 11:41:03 2018] Lustre: Skipped 4805832 previous similar messages [Fri Jul 6 11:51:03 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 11:51:03 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 7698222 previous similar messages [Fri Jul 6 11:51:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 11:51:03 2018] Lustre: Skipped 5080647 previous similar messages [Fri Jul 6 11:51:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 11:51:03 2018] Lustre: Skipped 5079803 previous similar messages [Fri Jul 6 12:00:37 2018] LustreError: 161083:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8805c78c4b00 x1605157378978848/t0(0) o37->049ed3af-b69e-859f-b357-9c6b4500ffed@172.16.229.39@o2ib:437/0 lens 568/440 e 0 to 0 dl 1530842212 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 12:00:37 2018] LustreError: 161083:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 22 previous similar messages [Fri Jul 6 12:00:46 2018] Lustre: 60612:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1530842208/real 1530842208] req@ffff880422531500 x1601781856586240/t0(0) o104->lustre-MDT0000@172.16.229.39@o2ib:15/16 lens 296/224 e 0 to 1 dl 1530842215 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Jul 6 12:00:46 2018] Lustre: 60612:0:(client.c:2100:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [Fri Jul 6 12:01:03 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 12:01:03 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 8159598 previous similar messages [Fri Jul 6 12:01:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 12:01:03 2018] Lustre: Skipped 6206327 previous similar messages [Fri Jul 6 12:01:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 12:01:03 2018] Lustre: Skipped 6204982 previous similar messages [Fri Jul 6 12:01:35 2018] LNetError: 2851:0:(o2iblnd_cb.c:3251:kiblnd_check_txs_locked()) Timed out tx: active_txs, 2 seconds [Fri Jul 6 12:01:35 2018] LNetError: 2851:0:(o2iblnd_cb.c:3326:kiblnd_check_conns()) Timed out RDMA with 172.16.229.39@o2ib (56): c: 4, oc: 0, rc: 8 [Fri Jul 6 12:01:42 2018] Lustre: 60612:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1530842265/real 1530842265] req@ffff880422531500 x1601781856586240/t0(0) o104->lustre-MDT0000@172.16.229.39@o2ib:15/16 lens 296/224 e 0 to 1 dl 1530842272 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Jul 6 12:01:42 2018] Lustre: 60612:0:(client.c:2100:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [Fri Jul 6 12:01:48 2018] LNet: 2851:0:(o2iblnd_cb.c:3297:kiblnd_check_conns()) Timed out tx for 172.16.229.39@o2ib: 15 seconds [Fri Jul 6 12:01:48 2018] LNet: 2851:0:(o2iblnd_cb.c:3297:kiblnd_check_conns()) Skipped 13 previous similar messages [Fri Jul 6 12:02:26 2018] LNet: 2851:0:(o2iblnd_cb.c:3297:kiblnd_check_conns()) Timed out tx for 172.16.229.39@o2ib: 0 seconds [Fri Jul 6 12:02:51 2018] LNet: 2851:0:(o2iblnd_cb.c:3297:kiblnd_check_conns()) Timed out tx for 172.16.229.39@o2ib: 25 seconds [Fri Jul 6 12:02:51 2018] LNet: 2851:0:(o2iblnd_cb.c:3297:kiblnd_check_conns()) Skipped 2 previous similar messages [Fri Jul 6 12:02:58 2018] Lustre: 60612:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1530842341/real 1530842341] req@ffff880422531500 x1601781856586240/t0(0) o104->lustre-MDT0000@172.16.229.39@o2ib:15/16 lens 296/224 e 0 to 1 dl 1530842348 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Jul 6 12:02:58 2018] Lustre: 60612:0:(client.c:2100:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [Fri Jul 6 12:03:29 2018] LNet: 2851:0:(o2iblnd_cb.c:3297:kiblnd_check_conns()) Timed out tx for 172.16.229.39@o2ib: 0 seconds [Fri Jul 6 12:03:29 2018] LNet: 2851:0:(o2iblnd_cb.c:3297:kiblnd_check_conns()) Skipped 2 previous similar messages [Fri Jul 6 12:03:59 2018] LNet: Service thread pid 60612 was inactive for 200.65s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Jul 6 12:03:59 2018] LNet: Skipped 1 previous similar message [Fri Jul 6 12:03:59 2018] Pid: 60612, comm: mdt00_014 [Fri Jul 6 12:03:59 2018] Call Trace: [Fri Jul 6 12:03:59 2018] [] schedule+0x29/0x70 [Fri Jul 6 12:03:59 2018] [] schedule_timeout+0x174/0x2c0 [Fri Jul 6 12:03:59 2018] [] ? process_timeout+0x0/0x10 [Fri Jul 6 12:03:59 2018] [] ptlrpc_set_wait+0x208/0x7a0 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] ? default_wake_function+0x0/0x20 [Fri Jul 6 12:03:59 2018] [] ldlm_run_ast_work+0xd3/0x3a0 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] ldlm_handle_conflict_lock+0x75/0x330 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] ldlm_process_inodebits_lock+0x151/0x490 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] ldlm_lock_enqueue+0x41b/0x8f0 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] ? ldlm_lock_create+0x1fc/0xa30 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] ldlm_cli_enqueue_local+0x1c3/0x850 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] ? kiblnd_send+0x357/0xa10 [ko2iblnd] [Fri Jul 6 12:03:59 2018] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] [Fri Jul 6 12:03:59 2018] [] mdt_object_local_lock+0x4d1/0xae0 [mdt] [Fri Jul 6 12:03:59 2018] [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] [Fri Jul 6 12:03:59 2018] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] ? lod_xattr_get+0xeb/0x6f0 [lod] [Fri Jul 6 12:03:59 2018] [] mdt_object_lock_internal+0x70/0x330 [mdt] [Fri Jul 6 12:03:59 2018] [] mdt_reint_object_lock+0x2c/0x60 [mdt] [Fri Jul 6 12:03:59 2018] [] mdt_reint_setattr+0x69c/0x10a0 [mdt] [Fri Jul 6 12:03:59 2018] [] mdt_reint_rec+0x80/0x210 [mdt] [Fri Jul 6 12:03:59 2018] [] mdt_reint_internal+0x5fb/0x9c0 [mdt] [Fri Jul 6 12:03:59 2018] [] mdt_reint+0x67/0x140 [mdt] [Fri Jul 6 12:03:59 2018] [] tgt_request_handle+0x925/0x13b0 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] ? default_wake_function+0x12/0x20 [Fri Jul 6 12:03:59 2018] [] ? __wake_up_common+0x58/0x90 [Fri Jul 6 12:03:59 2018] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [Fri Jul 6 12:03:59 2018] [] kthread+0xcf/0xe0 [Fri Jul 6 12:03:59 2018] [] ? kthread+0x0/0xe0 [Fri Jul 6 12:03:59 2018] [] ret_from_fork+0x58/0x90 [Fri Jul 6 12:03:59 2018] [] ? kthread+0x0/0xe0 [Fri Jul 6 12:03:59 2018] LustreError: dumping log to /tmp/lustre-log.1530842409.60612 [Fri Jul 6 12:04:02 2018] LNet: Service thread pid 4378 was inactive for 200.09s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Jul 6 12:04:02 2018] Pid: 4378, comm: mdt01_000 [Fri Jul 6 12:04:02 2018] Call Trace: [Fri Jul 6 12:04:02 2018] [] schedule+0x29/0x70 [Fri Jul 6 12:04:02 2018] [] schedule_timeout+0x174/0x2c0 [Fri Jul 6 12:04:02 2018] [] ? process_timeout+0x0/0x10 [Fri Jul 6 12:04:02 2018] [] ? ptlrpc_interrupted_set+0x0/0x110 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ptlrpc_set_wait+0x208/0x7a0 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ? default_wake_function+0x0/0x20 [Fri Jul 6 12:04:02 2018] [] ldlm_run_ast_work+0xd3/0x3a0 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ldlm_handle_conflict_lock+0x75/0x330 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ldlm_process_inodebits_lock+0x151/0x490 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ldlm_lock_enqueue+0x41b/0x8f0 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ? ldlm_lock_create+0x1fc/0xa30 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ldlm_cli_enqueue_local+0x1c3/0x850 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] [Fri Jul 6 12:04:02 2018] [] mdt_object_local_lock+0x4d1/0xae0 [mdt] [Fri Jul 6 12:04:02 2018] [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] [Fri Jul 6 12:04:02 2018] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ? lod_xattr_get+0xeb/0x6f0 [lod] [Fri Jul 6 12:04:02 2018] [] mdt_object_lock_internal+0x70/0x330 [mdt] [Fri Jul 6 12:04:02 2018] [] mdt_reint_object_lock+0x2c/0x60 [mdt] [Fri Jul 6 12:04:02 2018] [] mdt_reint_setattr+0x69c/0x10a0 [mdt] [Fri Jul 6 12:04:02 2018] [] mdt_reint_rec+0x80/0x210 [mdt] [Fri Jul 6 12:04:02 2018] [] mdt_reint_internal+0x5fb/0x9c0 [mdt] [Fri Jul 6 12:04:02 2018] [] mdt_reint+0x67/0x140 [mdt] [Fri Jul 6 12:04:02 2018] [] tgt_request_handle+0x925/0x13b0 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ? default_wake_function+0x12/0x20 [Fri Jul 6 12:04:02 2018] [] ? __wake_up_common+0x58/0x90 [Fri Jul 6 12:04:02 2018] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [Fri Jul 6 12:04:02 2018] [] kthread+0xcf/0xe0 [Fri Jul 6 12:04:02 2018] [] ? kthread+0x0/0xe0 [Fri Jul 6 12:04:02 2018] [] ret_from_fork+0x58/0x90 [Fri Jul 6 12:04:02 2018] [] ? kthread+0x0/0xe0 [Fri Jul 6 12:04:02 2018] LustreError: dumping log to /tmp/lustre-log.1530842412.4378 [Fri Jul 6 12:04:08 2018] Lustre: MGS: haven't heard from client bbf024dd-4ef2-2de6-6d80-0c3587f7b627 (at 172.16.229.39@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a47829000, cur 1530842418 expire 1530842268 last 1530842191 [Fri Jul 6 12:04:22 2018] Lustre: lustre-MDT0000: haven't heard from client 049ed3af-b69e-859f-b357-9c6b4500ffed (at 172.16.229.39@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ff1e2e000, cur 1530842432 expire 1530842282 last 1530842205 [Fri Jul 6 12:04:32 2018] LNet: Service thread pid 4378 completed after 229.92s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Fri Jul 6 12:04:32 2018] LNet: Skipped 1 previous similar message [Fri Jul 6 12:04:44 2018] LNet: 2851:0:(o2iblnd_cb.c:3297:kiblnd_check_conns()) Timed out tx for 172.16.229.39@o2ib: 0 seconds [Fri Jul 6 12:04:44 2018] LNet: 2851:0:(o2iblnd_cb.c:3297:kiblnd_check_conns()) Skipped 5 previous similar messages [Fri Jul 6 12:04:44 2018] LNet: Service thread pid 60612 completed after 245.56s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Fri Jul 6 12:11:03 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 12:11:03 2018] LustreError: 42073:0:(lod_dev.c:1414:lod_sync()) Skipped 7829007 previous similar messages [Fri Jul 6 12:11:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 12:11:03 2018] Lustre: Skipped 5199106 previous similar messages [Fri Jul 6 12:11:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 12:11:03 2018] Lustre: Skipped 5198706 previous similar messages [Fri Jul 6 12:21:03 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 12:21:03 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 6913210 previous similar messages [Fri Jul 6 12:21:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 12:21:03 2018] Lustre: Skipped 4467926 previous similar messages [Fri Jul 6 12:21:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 12:21:03 2018] Lustre: Skipped 4466344 previous similar messages [Fri Jul 6 12:31:03 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 12:31:03 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) Skipped 6133889 previous similar messages [Fri Jul 6 12:31:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 12:31:03 2018] Lustre: Skipped 3938022 previous similar messages [Fri Jul 6 12:31:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 12:31:03 2018] Lustre: Skipped 3937544 previous similar messages [Fri Jul 6 12:41:03 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 12:41:03 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 7368735 previous similar messages [Fri Jul 6 12:41:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 12:41:03 2018] Lustre: Skipped 4916430 previous similar messages [Fri Jul 6 12:41:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 12:41:03 2018] Lustre: Skipped 4915714 previous similar messages [Fri Jul 6 12:51:03 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 12:51:03 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 7634442 previous similar messages [Fri Jul 6 12:51:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 12:51:03 2018] Lustre: Skipped 5058397 previous similar messages [Fri Jul 6 12:51:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 12:51:03 2018] Lustre: Skipped 5056990 previous similar messages [Fri Jul 6 13:01:03 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 13:01:03 2018] LustreError: 99661:0:(lod_dev.c:1414:lod_sync()) Skipped 8573103 previous similar messages [Fri Jul 6 13:01:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 13:01:03 2018] Lustre: Skipped 5329083 previous similar messages [Fri Jul 6 13:01:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 13:01:03 2018] Lustre: Skipped 5328448 previous similar messages [Fri Jul 6 13:11:03 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 13:11:03 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 8492239 previous similar messages [Fri Jul 6 13:11:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 13:11:03 2018] Lustre: Skipped 5556207 previous similar messages [Fri Jul 6 13:11:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 13:11:03 2018] Lustre: Skipped 5555457 previous similar messages [Fri Jul 6 13:21:03 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 13:21:03 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 8637103 previous similar messages [Fri Jul 6 13:21:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 13:21:03 2018] Lustre: Skipped 5573042 previous similar messages [Fri Jul 6 13:21:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 13:21:03 2018] Lustre: Skipped 5572126 previous similar messages [Fri Jul 6 13:31:03 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 13:31:03 2018] LustreError: 5513:0:(lod_dev.c:1414:lod_sync()) Skipped 7064579 previous similar messages [Fri Jul 6 13:31:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 13:31:03 2018] Lustre: Skipped 4741120 previous similar messages [Fri Jul 6 13:31:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 13:31:03 2018] Lustre: Skipped 4740623 previous similar messages [Fri Jul 6 13:41:03 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 13:41:03 2018] LustreError: 8508:0:(lod_dev.c:1414:lod_sync()) Skipped 7103164 previous similar messages [Fri Jul 6 13:41:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 13:41:03 2018] Lustre: Skipped 4724697 previous similar messages [Fri Jul 6 13:41:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 13:41:03 2018] Lustre: Skipped 4724279 previous similar messages [Fri Jul 6 13:51:03 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 13:51:03 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) Skipped 7388804 previous similar messages [Fri Jul 6 13:51:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 13:51:03 2018] Lustre: Skipped 4961684 previous similar messages [Fri Jul 6 13:51:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 13:51:03 2018] Lustre: Skipped 4960516 previous similar messages [Fri Jul 6 14:01:03 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 14:01:03 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 7053181 previous similar messages [Fri Jul 6 14:01:03 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 14:01:03 2018] Lustre: Skipped 4849273 previous similar messages [Fri Jul 6 14:01:03 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 14:01:03 2018] Lustre: Skipped 4848485 previous similar messages [Fri Jul 6 14:11:03 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 14:11:03 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 6146850 previous similar messages [Fri Jul 6 14:11:04 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 14:11:04 2018] Lustre: Skipped 4132935 previous similar messages [Fri Jul 6 14:11:04 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 14:11:04 2018] Lustre: Skipped 4132077 previous similar messages [Fri Jul 6 14:21:04 2018] LustreError: 99640:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 14:21:04 2018] LustreError: 99640:0:(lod_dev.c:1414:lod_sync()) Skipped 7302626 previous similar messages [Fri Jul 6 14:21:04 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 14:21:04 2018] Lustre: Skipped 4478087 previous similar messages [Fri Jul 6 14:21:04 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 14:21:04 2018] Lustre: Skipped 4477648 previous similar messages [Fri Jul 6 14:31:04 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 14:31:04 2018] LustreError: 172046:0:(lod_dev.c:1414:lod_sync()) Skipped 8893271 previous similar messages [Fri Jul 6 14:31:04 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 14:31:04 2018] Lustre: Skipped 5737538 previous similar messages [Fri Jul 6 14:31:04 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 14:31:04 2018] Lustre: Skipped 5736862 previous similar messages [Fri Jul 6 14:41:04 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 14:41:04 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 8270266 previous similar messages [Fri Jul 6 14:41:04 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 14:41:04 2018] Lustre: Skipped 5517738 previous similar messages [Fri Jul 6 14:41:04 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 14:41:04 2018] Lustre: Skipped 5515309 previous similar messages [Fri Jul 6 14:51:04 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 14:51:04 2018] LustreError: 5212:0:(lod_dev.c:1414:lod_sync()) Skipped 8314571 previous similar messages [Fri Jul 6 14:51:04 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 14:51:04 2018] Lustre: Skipped 5482851 previous similar messages [Fri Jul 6 14:51:04 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 14:51:04 2018] Lustre: Skipped 5481442 previous similar messages [Fri Jul 6 15:01:04 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 15:01:04 2018] LustreError: 76898:0:(lod_dev.c:1414:lod_sync()) Skipped 8406990 previous similar messages [Fri Jul 6 15:01:04 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 15:01:04 2018] Lustre: Skipped 5358530 previous similar messages [Fri Jul 6 15:01:04 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 15:01:04 2018] Lustre: Skipped 5358002 previous similar messages [Fri Jul 6 15:11:04 2018] LustreError: 60612:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 15:11:04 2018] LustreError: 60612:0:(lod_dev.c:1414:lod_sync()) Skipped 8842777 previous similar messages [Fri Jul 6 15:11:04 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 15:11:04 2018] Lustre: Skipped 5643690 previous similar messages [Fri Jul 6 15:11:04 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 15:11:04 2018] Lustre: Skipped 5642897 previous similar messages [Fri Jul 6 15:21:04 2018] LustreError: 60612:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 15:21:04 2018] LustreError: 60612:0:(lod_dev.c:1414:lod_sync()) Skipped 7650035 previous similar messages [Fri Jul 6 15:21:04 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 15:21:04 2018] Lustre: Skipped 4854736 previous similar messages [Fri Jul 6 15:21:04 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 15:21:04 2018] Lustre: Skipped 4854384 previous similar messages [Fri Jul 6 15:31:04 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 15:31:04 2018] LustreError: 6672:0:(lod_dev.c:1414:lod_sync()) Skipped 8696642 previous similar messages [Fri Jul 6 15:31:04 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 15:31:04 2018] Lustre: Skipped 5660399 previous similar messages [Fri Jul 6 15:31:04 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 15:31:04 2018] Lustre: Skipped 5659742 previous similar messages [Fri Jul 6 15:35:03 2018] LustreError: 73114:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffe311850 x1603048946389248/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:504/0 lens 568/440 e 0 to 0 dl 1530855114 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 15:35:03 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.230.53@o2ib [Fri Jul 6 15:35:03 2018] LNet: 2852:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 7 previous similar messages [Fri Jul 6 15:35:03 2018] LustreError: 73114:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 15 previous similar messages [Fri Jul 6 15:35:04 2018] LustreError: 73114:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880419292a00 x1603048946454400/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:468/0 lens 568/440 e 0 to 0 dl 1530855078 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 15:35:04 2018] LustreError: 73114:0:(ldlm_lib.c:3178:target_bulk_io()) Skipped 323 previous similar messages [Fri Jul 6 15:35:09 2018] LustreError: 73113:0:(ldlm_lib.c:3178:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880a976c9e00 x1603048946782080/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:473/0 lens 568/440 e 0 to 0 dl 1530855083 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 15:35:09 2018] LustreError: 73114:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880c3e6e7500 x1603048946782080/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:473/0 lens 568/440 e 0 to 0 dl 1530855083 ref 1 fl Interpret:/2/0 rc 0/0 [Fri Jul 6 15:35:09 2018] LustreError: 73114:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 4 previous similar messages [Fri Jul 6 15:35:09 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.230.53@o2ib [Fri Jul 6 15:35:09 2018] LNet: 2855:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) Skipped 1 previous similar message [Fri Jul 6 15:41:04 2018] LustreError: 60612:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 15:41:04 2018] LustreError: 60612:0:(lod_dev.c:1414:lod_sync()) Skipped 8521349 previous similar messages [Fri Jul 6 15:41:04 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 15:41:04 2018] Lustre: Skipped 5438563 previous similar messages [Fri Jul 6 15:41:04 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 15:41:04 2018] Lustre: Skipped 5437914 previous similar messages [Fri Jul 6 15:51:04 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 15:51:04 2018] LustreError: 70058:0:(lod_dev.c:1414:lod_sync()) Skipped 7181724 previous similar messages [Fri Jul 6 15:51:04 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 15:51:04 2018] Lustre: Skipped 4723854 previous similar messages [Fri Jul 6 15:51:04 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 15:51:04 2018] Lustre: Skipped 4723319 previous similar messages [Fri Jul 6 16:01:04 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 16:01:04 2018] LustreError: 58730:0:(lod_dev.c:1414:lod_sync()) Skipped 8703599 previous similar messages [Fri Jul 6 16:01:04 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 16:01:04 2018] Lustre: Skipped 5695677 previous similar messages [Fri Jul 6 16:01:04 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 16:01:04 2018] Lustre: Skipped 5695038 previous similar messages [Fri Jul 6 16:11:04 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 16:11:04 2018] LustreError: 42072:0:(lod_dev.c:1414:lod_sync()) Skipped 8586709 previous similar messages [Fri Jul 6 16:11:04 2018] Lustre: lustre-MDT0000: Client 47912236-9591-81de-cfda-439b0686f05e (at 172.16.230.53@o2ib) reconnecting [Fri Jul 6 16:11:04 2018] Lustre: Skipped 5291166 previous similar messages [Fri Jul 6 16:11:04 2018] Lustre: lustre-MDT0000: Connection restored to a4c78123-b9f1-ed44-935a-eeb427513e70 (at 172.16.230.53@o2ib) [Fri Jul 6 16:11:04 2018] Lustre: Skipped 5290614 previous similar messages [Fri Jul 6 16:11:30 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8809a1e9a100 x1603049074120192/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:389/0 lens 568/440 e 0 to 0 dl 1530857264 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 16:11:30 2018] LNet: 2854:0:(o2iblnd_cb.c:398:kiblnd_handle_rx()) PUT_NACK from 172.16.230.53@o2ib [Fri Jul 6 16:11:30 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 2 previous similar messages [Fri Jul 6 16:11:32 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880ffd2fb050 x1603049074253472/t0(0) o37->47912236-9591-81de-cfda-439b0686f05e@172.16.230.53@o2ib:428/0 lens 568/440 e 0 to 0 dl 1530857303 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Jul 6 16:11:32 2018] LustreError: 159530:0:(ldlm_lib.c:3229:target_bulk_io()) Skipped 2 previous similar messages [Fri Jul 6 16:21:04 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 16:21:04 2018] LustreError: 172047:0:(lod_dev.c:1414:lod_sync()) Skipped 7990384 previous similar messages [Fri Jul 6 16:21:04 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 16:21:04 2018] Lustre: Skipped 5220204 previous similar messages [Fri Jul 6 16:21:04 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 16:21:04 2018] Lustre: Skipped 5219229 previous similar messages [Fri Jul 6 16:31:04 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 16:31:04 2018] LustreError: 60611:0:(lod_dev.c:1414:lod_sync()) Skipped 9283873 previous similar messages [Fri Jul 6 16:31:04 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 16:31:04 2018] Lustre: Skipped 5987569 previous similar messages [Fri Jul 6 16:31:04 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 16:31:04 2018] Lustre: Skipped 5986290 previous similar messages [Fri Jul 6 16:41:04 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 16:41:04 2018] LustreError: 89767:0:(lod_dev.c:1414:lod_sync()) Skipped 10065580 previous similar messages [Fri Jul 6 16:41:04 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 16:41:04 2018] Lustre: Skipped 6627051 previous similar messages [Fri Jul 6 16:41:04 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 16:41:04 2018] Lustre: Skipped 6625773 previous similar messages [Fri Jul 6 16:51:04 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 12: -107 [Fri Jul 6 16:51:04 2018] LustreError: 181962:0:(lod_dev.c:1414:lod_sync()) Skipped 8274044 previous similar messages [Fri Jul 6 16:51:04 2018] Lustre: lustre-MDT0000: Client 66471b3c-6a3e-724d-5030-ee8252fcfcd2 (at 172.16.230.87@o2ib) reconnecting [Fri Jul 6 16:51:04 2018] Lustre: Skipped 5378430 previous similar messages [Fri Jul 6 16:51:04 2018] Lustre: lustre-MDT0000: Connection restored to 8e150630-2b04-00b6-c100-b58229b19cac (at 172.16.230.87@o2ib) [Fri Jul 6 16:51:04 2018] Lustre: Skipped 5377677 previous similar messages